jensjepsen/danish-lm-400m-icl-v3

danish-lm-400m-icl-v3

400M Danish model tuned for in-context schema and format induction: given a few worked examples in a single user turn and no instruction, infer both the output schema and the output format from the examples and apply them to a new passage.

Recipe

• —Base: `jensjepsen/danish-lm-400m-sft-v31-avg-top3`
• —Data: `jensjepsen/danish-icl-schema-format-v3` sft:train, 33,933 rows, ICL-only, no ballast
• —3 epochs, 3,024 steps, effbs 32 (16 x 2), lr 1e-5 constant + 50 warmup, `adamwbnb_8bit, seq 3072, FA2 varlen (DataCollatorWithFlattening`)
• —1,208s on one RTX 5090. Tokenizer is the base checkpoint's (16007 tokens, chat tokens at 16000-16002).

Evals

Exact match on the parsed object, greedy, n=400 per split. Base = the v31 checkpoint this was tuned from, measured in the same session.

Trained formats on unseen schemas are even: numbered 51.7, tsv 43.8, tagged 43.6, kvcolon 41.0, json 38.7, kvbracket 36.5, kv_arrow 35.5.

Meaning-free keys are about as easy as real Danish field names (val 76.7 vs 78.7; eval_schema 40.1 vs 42.9), which is the evidence the mapping is read from the demonstrations rather than from field-name semantics — the base scored 0.6% on symbol rows.

Limits

Format transfer is delimiter substitution, not structural generalisation. On unseen formats, kv_eq (k=v, one delimiter away from trained formats) reaches 78.5%, while bracket_pair ([k]v[/k]) scores 0.0% and brace_pair ({k}v{/k}) 0.7% — despite bracket_pair being structurally identical to the trained tagged (<k>v</k>). The model copies the opening delimiter ~100% of the time and gets every value right ~60% of the time, but produces the matched close only ~37%, falling back to the trained </tag> form. To get a format into the repertoire, train on it.

Out-of-distribution tasks work, limited by task ability not format. On daneplus NER (never trained on; 3-shot, no instruction) it parses 100% in `tagged` and `kvcolon` against the base's 52%/24%, reaching 26-31 entity F1 where the base gets 0-2.4. Residual errors are entity typing, not format.

Trained ICL-only. The predecessor trained on a single format became rigid about structured output; multi-format training was the fix. General benchmarks were roughly flat but drifted down over the run (gsm8k 17.0 to 12.5 at n=200, citgen 28.5 to 24.0) — mix in ballast if that matters for your use.

Prompt shape

Exemplars go inside one user turn. This model family breaks on multi-turn few-shot (the v31 card records GSM8K 18.7 to 2.1%).

<|user|>Eksempler:

Tekst:
<passage>
Svar: <rendered answer>

Tekst:
<passage>
Svar: <rendered answer>

Tekst:
<target passage>
Svar:<|end|><|assistant|>