richardlian/iolai-2026-4b-direct-vote5

iolai-2026-4b-direct-vote5 — IOL-AI 2026 solver (best measured configuration)

Hidden-set score: 0.1453 (EM 0.0935, chrF 0.2259, 90/90 rows delivered). That is the best of the ten configurations we submitted, and it beat the 12-July public leaderboard top (0.147 EM 0.2592 / chrF 0.0833) on chrF by ~2.7× while running a 4B model on a T4.

This repo is a complete, self-contained submission: script.py reads /tmp/data/test.csv and writes submission.csv, with no network access and no runtime installs. Everything below is meant to be forked and extended — the solver is deliberately structured so that adding a step, a prompt, or a retrieval corpus is a small, local change.

Our measured results (all on the same hidden test set)

The two findings that produced most of the score:

1. 1.Self-consistency voting roughly doubled EM twice (0.0139 → 0.0623 → 0.0935). Sampling the same problem several times and taking a per-item majority is worth far more than any model or prompt change we tried.
2. 2.Delivery, not reasoning, was the early bottleneck. A categorized error analysis of the remaining misses found 86% are genuinely wrong answers and 0% are formatting/diacritic failures — so once output is reliable, the gap is model capability, and the honest lever is better use of the samples you can afford.

How it works

script.py sets a recipe in the environment and calls v2.script.main(). Order of operations is load-bearing:

1. 1.Budget clock starts before any import. The platform hard-kills at 30 minutes.
2. 2.A valid `submission.csv` is written before the model loads — every row filled with an in-domain token from its own puzzle text. Later stages overwrite it via write-then-rename. A crash, an OOM, or the kill can never leave a blank row, and blank rows forfeit the chrF partial credit that is half the metric.
3. 3.Passes over all problems: greedy solve → finalize (rescue truncated output) → validate → refine → vote → finalize. Every pass re-aggregates and re-flushes, so partial credit is banked continuously instead of at the end.
4. 4.Aggregation parses each sample, votes per item, and fills any remaining blank from the fallback.

Parsing and voting

Answers are emitted as a bare JSON list (the model is fine-tuned to do this), parsed by scanning for the last [ that json.JSONDecoder().raw_decode accepts — a regex cannot do this correctly because ] occurs inside quoted answers. Voting is per item over a normalization (case, surrounding quotes, trailing period) with the surface form taken from the first sample that proposed the winner.

Parallel vote decoding (in the vote12 sibling repo, and available here)

Voting N times naively costs N prefills. Two facts fix that:

• —The sandbox's seccomp filter blocks every socket syscall — TCP loopback, AF_UNIX, socketpair all return EPERM — so llama-server and every HTTP inference server are unusable. Pipes survive.
• —All N votes for a problem share one prompt, so they should share one prefill.

bin/llama-iolbatch is a small llama.cpp example that speaks JSONL over stdin/stdout: one request carries n_samples, the driver prefills the prompt once, clones the KV prefix to sibling sequences with llama_memory_seq_cp, and continuous-batch decodes them on a unified KV pool. Measured: 160 problems × 12 votes in 464 s in the replica.

Confidence-weighted voting (newer arms)

The driver can also return the raw-model logprob of every emitted token plus that token's byte length, which lets the Python side map each answer substring back to the tokens that produced it and weight votes by per-item confidence. On our local 160-problem fixture this is a real gain over plain majority (paired Δgeo +0.0197, 95% CI [+0.0042, +0.0341]) while plain majority is flat in N. Sibling repos carry that arm; see v2/confidence.py.

Using this in your own submission

git clone https://huggingface.co/richardlian/iolai-2026-4b-direct-vote5

Layout: script.py (recipe) · v2/ (solver package) · model/ (Q8_0 GGUF) · vendor/ (llama-cpp-python + deps, cp310) · bin/ (the batch driver).

Every knob is an environment variable read by v2/config.py:

N_SAMPLES=5          # votes per problem
MAX_NEW=700          # generation cap per sample
VOTE_TEMP=1.0        # vote sampling temperature
VOTE_WEIGHT=none     # none|tiebreak|full|bestof (confidence weighting)
VOTE_LP_T=0.35       # weight temperature for the weighted modes
PROMPT_MODE=direct   # trained direct-JSON answers
BACKEND=parallel     # use bin/llama-iolbatch instead of the bindings
SLOTS=16             # concurrent decode streams
LLAMA_CTX=16384      # must cover SLOTS x (prompt + MAX_NEW)
TIME_LIMIT_S=1800    # the platform's hard kill

Budget arithmetic — do this before changing anything

MAX_NEW × n_rows ÷ measured_tokens_per_second must fit inside TIME_LIMIT_S. Measured on a real T4: this 4B at Q8 = 40.5 tok/s single-stream (5.5 s load, 5.0 GB VRAM); Gemma-4-26B-A4B at IQ4XS = 49.5 tok/s, 14.7 GB. We once shipped a reasoning-length recipe (~1400 tokens/row) that did not fit, and it scored **0.0** — every row truncated before its answer. The solver degrades gracefully (it shrinks `MAXNEW` under time pressure rather than abandoning rows), but it cannot rescue a recipe that was never affordable.

Adding a pipeline step

A Pass is anything with a name and run(states, ctx):

class MyPass:
    name = "mypass"

    def run(self, states, ctx):
        for s in states:
            if ctx.budget.exhausted():      # always ask the budget first
                return
            texts = ctx.backend.chat([conv], greedy=False, max_new=256)
            s.samples.append({"kind": "solve", "text": texts[0], "meta": None})

Add it to v2/pipeline.py::default_passes() and it inherits transcript logging, per-pass flushing, crash-safety, and the deadline guard.

Adding retrieval (RAG)

There is no network in the sandbox, so retrieval means shipping a corpus in the repo and querying it locally (a pickled inverted index, a small embedding matrix in .npy, or plain files). Where the retrieved text goes matters more than how you fetch it:

• —Put it in a REFERENCE block inside the USER turn, never in the system prompt. This model was fine-tuned with a fixed system prompt; changing it moves the model off-distribution. We measured that directly: swapping the system prompt on an untuned model collapsed delivery to 9 of 159 rows.
• —Keep it to ~1500 tokens or less. Retrieved text is prefill, charged once per problem, and it competes with decode for the same 30 minutes.
• —Re-validate the budget and the context. LLAMA_CTX must still cover SLOTS × (prompt + MAX_NEW); the driver is started with --fit off so it fails loudly rather than silently shrinking the context and starving votes.

# v2/prompts.py — extend the user turn, leave SYSTEM alone
def solve_messages(context, query, *, fewshot, reference=""):
    user = f"{context}\n\n{query}"
    if reference:
        user = f"REFERENCE (may help; ignore if irrelevant):\n{reference}\n\n{user}"
    return [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]

Good corpora for this task are typological facts, IPA charts, and numeral-system descriptions — knowledge a 4B model lacks and cannot derive from five examples. Since 86% of our residual errors are genuinely wrong answers, buying capability is the right target.

Hard-won lessons (each cost us a submission)

• —A parser change is not a parser change. A bundle of output-alignment fixes was neutral on our local fixture, doubled the score of one model (0.0437 → 0.0866), and destroyed another (0.0883 → 0.0312) — because part of it fed an inflated item count back into generation. Anything that touches what the model generates must be A/B'd per model on the hidden set, not just on a local fixture.
• —Verify that a fine-tune actually merged. Loading Qwen3.5-4B with AutoModelForCausalLM gives the vision wrapper (text stack at model.language_model.*) while adapters trained via unsloth key model.model.*. PEFT warns about missing adapter keys, merges nothing, and ships the base model. Three of our submissions were base models before we caught it. Always assert weights changed (not torch.allclose) and keep a base-model control.
• —Early stopping on eval loss underfits this task. Format acquisition continues after cross-entropy plateaus; selecting checkpoints by generated-output score was the only regularizer that worked.
• —Fine-tuning is not universally good. The same trace-SFT that lifted the 4B 3.75× made an already-capable 26B worse at every dose we tried.
• —Explanations should be empty if you have none. The optional explanation column invites boilerplate; a canned sentence is not an explanation and inflates "explanation rate" to a meaningless 100%.

Model and licensing

Qwen3.5-4B (Apache-2.0) fine-tuned with LoRA on 239 IOL-style problems with byte-verbatim answers in a direct-JSON output format, merged and quantized to Q8_0 (≈lossless; the 4B has no memory pressure on a 16 GB T4). Training data was expert-verified reasoning traces from our lab's corpus plus openly-licensed worked solutions; no LINGOLY/LingOly-TOO data was used (their terms forbid training).