teaguesterling/qwen3.5-0.8b-astcss-t5

qwen3.5-0.8b-astcss-t5

English → astcss selector (sittingduck's `astselect`) at 0.8B, trained on the tier-5 corpus.

This is the successor to `qwen3.5-0.8b-astcss`. Same model size, same card, same recipe — only the training corpus changed (820 pairs / 2,205 rows → 6,053 pairs / 15,928 rows / 2,599 distinct selector shapes).

Scores

Measured on this merged artifact, not carried over from the adapter — an fp16 merge is verified, not assumed. Both numbers reproduce the adapter tier-for-tier.

Scoring is by execution: the selector is run against the fixture and the returned node set is compared to the reference. A different string selecting the same nodes counts as correct — which matters, since exact-string agreement is only 56.4 % on eval_t5.

For scale on that second eval: a 4 B model trained on the old corpus scores 67.3 %, and an untuned 9 B scores 60.0 %. Tier-5 training is worth more here than either.

The 108-pair eval is saturated at this model size — the old corpus, this corpus at epoch 1 and at epoch 2 all land on exactly 89/108. Only the tier distribution moves. Read eval_t5 for the difference between these two models.

It needs the card

This model scores 0.0 % without its system prompt, at every model size we have tested. It was trained with a per-language vocabulary card in the system position; without one it answers as a chat model rather than emitting a selector. Cards live in the astcss-eval repo under train/cards/card_<lang>.md.

Greedy decoding, thinking off, ~6 completion tokens per answer, ~0.24 s median latency.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

m = "teaguesterling/qwen3.5-0.8b-astcss-t5"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.float16, device_map="cuda")

card = open("train/cards/card_python.md").read()   # REQUIRED
msgs = [{"role": "system", "content": card},
        {"role": "user", "content": "functions that call range without any try block"}]
ids = tok(tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False),
          return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=48, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
# .fn:has(.call#range):not(:has(.try))

What it is

LoRA (r=16, α=32, all-linear) on Qwen3.5-0.8B, fp16, 1 epoch, seed 18, per-language card, template cap 8. Merged into the base weights — no adapter needed at serving time. The merge was verified by tensor delta: 186 targeted tensors at max |Δ| 0.036 against 134 untouched at 0.0005, which is fp16 round-trip noise.

Epoch 1, deliberately. Epoch 2 had the better validation loss (0.0557 vs 0.0833) and the worse eval (80.0 % vs 83.6 % on eval_t5, identical on the 108-pair). Validation loss does not track the eval here.

Limitations

• —Text-only. save_pretrained under AutoModelForCausalLM drops the base's vision tower (153 model.visual.*) and multi-token-prediction head (15 mtp.*) — 168 base tensors are not written. If you need the multimodal stack, merge differently.
• —Needs the card, as above. This is not a general chat model.
• —Trained and evaluated on Python, Bash, C, C++, Go, Java, JavaScript, Rust and SQL fixtures.
• —eval_t5 is 55 pairs; a 2-pair difference there is within noise.