birgermoell/oellm-eu-defect-repair-sft-v1

oellm-eu-defect-repair-sft-v1

Monolingual SFT repair data for European-language generation defects observed after Qwen 2B/4B/9B post-training. This dataset is designed to repair degeneration, repetition loops, short answers, morphology damage, and Bulgarian/Russian language leakage.

Strict SFT Schema

Each row in data/*.jsonl uses exactly:

{"messages":[{"role":"user","content":"..."},{"role":"assistant","content":"..."}],"lang":"is"}

Provenance, source URL, license, source id, quality statistics, and hashes are kept in metadata/all_metadata.jsonl, source_registry.json, metadata/quality_report.json, and manifest.json.

Source Policy

Version v0.1.0 is built from native Wikipedia text via wikimedia/wikipedia 2023-11-01 snapshots. Rows are not raw machine translations. Prompt templates are short target-language instructions and assistant turns are human-authored native article passages.

License: Wikimedia text is redistributed under CC-BY-SA terms, with attribution and share-alike requirements. Keep the sidecar provenance files when using or redistributing filtered subsets.

Counts

• —bg: 2000
• —da: 3000
• —et: 8000
• —fi: 1000
• —ga: 8000
• —hr: 3000
• —hu: 3000
• —is: 8000
• —lt: 3000
• —lv: 3000
• —mt: 7352
• —pl: 1000
• —ro: 1000
• —sk: 3000
• —sl: 3000

Total strict SFT rows: 57352.

Intended Use

Use as an upweighted repair shard in SFT:

• —Tier 1 (is, ga, mt, et): high upweight for fluency and answer length.
• —Tier 2 (hr, sl, lt, lv, da, hu, sk): medium upweight for loop repair.
• —bg: anti-Russian leakage shard with Russian-marker filters.
• —Tier 4 (ro, pl, fi): light touch-up.

Run scripts/validate_defect_repair_sft.py --dataset-dir data/defect_repair/oellm-eu-defect-repair-sft-v1 before packing to parquet or mixing into SFT.