john/folksy_idioms
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
3 commits 1 branch 0 tags 6.3 MiB
Find a file
john 651ec3ffc6 Fix generator quality issues and run initial corpus pipeline 
Pre-corpus fixes (from EVALUATION.md):
- Clean 2,264 contaminated rows from augmented relations (bridge
  artifacts, full-sentence HasProperty values, null bytes, empty words)
- Fix article logic: dynamic a/an across Deconstruction, FalseEquivalence,
  DenialOfConsequences, TautologicalWisdom templates
- Tighten _short_concepts() default from max_words=3 to 2
- Fix FutilePreparation gerunding: filter vocab nouns and noun-suffix
  words from UsedFor targets; fix CVC doubling for 'y'-ending words
- Add _looks_like_verb() heuristic, improve _a() for vowel-sound edges

Pipeline hardening:
- polish_corpus.py: context-size fallback (truncate chain, then minimal
  prompt), classified error types, consecutive-error circuit breaker,
  10-entry flush granularity, ETA tracking, KeyboardInterrupt handling
- generate_raw_batch.sh: fix python -> python3

Corpus generation run (9,835 raw -> 5,499 polished -> 2,312 filtered):
- 44.1% discard rate, 0 errors, 82 minutes on RTX 4090
- 9,257 training pairs across 5 input framing types
- 97.6% vocab coverage (609/624 words)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 04:33:56 -04:00
corpus Fix generator quality issues and run initial corpus pipeline 2026-03-10 04:33:56 -04:00
data Fix generator quality issues and run initial corpus pipeline 2026-03-10 04:33:56 -04:00
examples Initial 'folksy idiom' generator 2026-02-15 14:04:25 -05:00
schemas Initial 'folksy idiom' generator 2026-02-15 14:04:25 -05:00
scripts Fix generator quality issues and run initial corpus pipeline 2026-03-10 04:33:56 -04:00
.gitignore corpus generation (work from mid february) 2026-03-09 19:52:09 -04:00
CORPUS_GENERATION_SPEC.md corpus generation (work from mid february) 2026-03-09 19:52:09 -04:00
EVALUATION.md corpus generation (work from mid february) 2026-03-09 19:52:09 -04:00
folksy_generator.py Fix generator quality issues and run initial corpus pipeline 2026-03-10 04:33:56 -04:00
FOLKSY_GENERATOR_SPEC.md Initial 'folksy idiom' generator 2026-02-15 14:04:25 -05:00
GRAPH_ENHANCEMENT_SPEC.md corpus generation (work from mid february) 2026-03-09 19:52:09 -04:00