hemlang/Schierling-Qwen3.6-27B-LoRA

Schierling-Qwen3.6-27B-LoRA

Schierling: German for hemlock.

A LoRA that teaches Hemlock — a systems scripting language with manual memory management — to Qwen3.6-27B. Supervised fine-tuning on 5,562 examples.

hembench: 10.1% → 65.0%. Every generated program is executed by the real interpreter and its stdout compared against expected output, so these are programs that actually run, not code that looks plausible.

Results

22 tasks fixed, 0 broken. Nothing the base could already do was lost.

The sharpest single number: generated code that fails to execute drops from 30 tasks to 8. The base does not write runnable Hemlock at all — its 10.1% comes almost entirely from L6, where the task supplies existing code to repair rather than asking for a program from scratch.

L2 (stdlib) is the weak spot at 1/5. Standard-library work needs exact function names and signatures — memorised API knowledge rather than transferable syntax. The stdlib portion of the training data (the formulary rows folded into codex3) is the smallest slice, and the per-task cap trimmed it further. More stdlib coverage is the obvious next improvement.

Why SFT and not DPO

An earlier attempt taught Hemlock with 400 ORPO preference pairs and did not work. Inspecting that dataset showed why: only 186 pairs differed in code — the other 214 differed in which tool was called, so the objective could be satisfied by fixing tool selection without emitting a single correct line of Hemlock. Preference optimisation sharpens choices among behaviours a model already has; it does not install a grammar it has never seen.

Same language, same base, two instruments: 400 preference pairs moved nothing, 5,562 supervised examples moved 55 points.

Training

Two filters the dataset cards do not apply:

• —150 `bench_overlap` rows dropped. Their task names collide with hembench scenarios. The codex3 card suggests keeping them for SFT — fine if you never benchmark on hembench, and invalidating if you do. Dropping them is what makes the 65.0% a held-out number.
• —Per-task cap of 4. Tasks recur ~6× (one row per source language, identical target), so uncapped, 270 programs would be seen far more often than everything else.

codex-SFT, codex2-SFT and formulary-SFT are not used separately — codex3 supersedes and merges all three.

A note on the loss curve

Training loss fell 0.863 → 0.231 across epoch 1, then stepped to 0.068 at the epoch boundary and stayed flat for the remaining 230+ steps. That is the shape of memorisation rather than learning, and it suggested the second epoch was wasted.

The benchmark says otherwise — 26/38 on held-out tasks is not what a memorised model produces. Still, 1 epoch is worth trying: it would halve a 14-hour run. No epoch-1 checkpoint was kept here to compare against, since Merlina's epoch-end save overwrites; set save_steps explicitly if you want intermediates.

Limitations

• —stdlib is weak (1/5). Expect to check API calls against the docs.
• —Not a tool-calling model. The training data is 5,562 rows of direct code generation with zero tool calls. If you stack this with an agent adapter, measure whether delegation survives.
• —Trained and evaluated only at r=32 on the TIES base; behaviour on other Qwen3.6 descendants is untested.
• —38 benchmark tasks is a small sample — level scores of 5/7 carry wide error bars even where the overall difference is unambiguous.