logic65/Whittle-Qwen-3.8-35B-A3B

<p align="center"><img src="https://huggingface.co/logic65/Whittle-Next-27B-A3B/resolve/main/whittle.svg" width="640" alt="Whittle"></p>

Whittle-Qwen-3.8-35B-A3B

### ☕ Support this work Whittle is built by one person on a grocery budget and rented GPU hours. This model is the first in the line whose memory carries knowledge; the next steps — more memory rows, fact-dense training data, the on-policy distillation — are compute we cannot currently pay for. If this research is useful to you, or you want to see it finished: [ko-fi.com/davida81328](https://ko-fi.com/davida81328). Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

A 35.1 B-parameter, ~3 B-active mixture-of-experts in the Qwen3.8-Flash-Next (qwen4_exp) format whose 10 B-parameter hashed n-gram memory is load-bearing — the first Whittle where zeroing the memory measurably hurts the model. Distilled from Qwen3.8-27B (thinking on), built on Whittle-Next-27B-A3B v4.4, with memory contents transferred from Qwen3.8-Flash-Next's own table. Runs on stock llama.cpp, no patches.

GGUFs (Q8_0 → Q3_K_M) are on [logic65/Whittle-Qwen-3.8-35B-A3B-GGUF](https://huggingface.co/logic65/Whittle-Qwen-3.8-35B-A3B-GGUF). This repo holds the full weights (model-*.safetensors, 35.1 B incl. the memory), the config/tokenizer, and every probe in eval/.

Which weights are which (26 Sep 2026). The root is agentfix2 (step 300): lw5 plus 300 steps that fix a multi-step tool-use failure (see Agent use below). Its predecessors are preserved unchanged: lw5 under bf16-lw5/, lw2 under bf16-lw2/ and the first release tbl1 under bf16-tbl1/. agentfix2 trades about half of lw5's memory dependence for reliable tool use; if the memory's contribution matters more to you than agent work, run bf16-lw5/.

Research preview, not a finished distillation. This model uses the Qwen3.8-Flash-Next architecture (qwen4_exp: hyper-connections, gated DeltaNet + attention, the 10 B n-gram memory), and its memory rows come from Qwen3.8-Flash-Next's own table — but the body has seen a few thousand training steps on 1,840 teacher traces (maths and code review) plus the memory transfer. It still needs a full distillation — broad-domain, on-policy, with the memory trained jointly on far more data — before it can be called complete or compared to Qwen3.8 on equal terms. What it already shows is that the architecture works end-to-end on stock llama.cpp and that the memory is load-bearing. Read Measured and Caveats before relying on it.

What is in the box

Agent use: the post-success rewrite loop (fixed in agentfix2)

An outside Claude Code evaluation (25 Sep) found the lw5 root repeating identical Write calls until the turn limit. We reproduced it with synthetic tool-use probes (24 items each, graded on the tool-call arguments, serving sampler, Q8_0) and traced it: with thinking on, after a tool result reports success, the model's next reasoning block restarts the task from the user's original request and does the work again. It only happens when the client does not send the earlier reasoning back (the chat template then shows those turns with an empty thinking block), and it grows with the amount of tool output in between.

agentfix2 is lw5 plus 300 training steps on 212 rejection-sampled episodes of lw5's own tool use — stop after a success, run the next step, write after a read, retry after an error; only replies our grader accepted — trained on the reply tokens only, 20 % of steps, the rest code corpus. Clients that keep reasoning_content in the history avoid the loop on either checkpoint: we measured llama.cpp's Anthropic endpoint (what Claude Code talks to), the Vercel AI SDK and LiteLLM all carrying it through.

Claude Code now runs. Claude Code sends its environment block as a system message after the user turn (and another after every tool result). The previous chat template raised System message must be at the beginning, so every Claude Code request failed; the template now renders a system message wherever it arrives. Conversations with one leading system message render byte-for-byte as before. Claude Code completed a write-then-read task end to end against llama-server's /v1/messages with this template.

Measured

All numbers below are ours, on the same machine with the same scripts as the v4.4 baseline; the replies and logs are in eval/.

The memory carries knowledge. "Memory gain" = held-out cross-entropy with the memory zeroed minus with it on, on rows never trained on; positive means the body needs the table. Every previous Whittle sat at ±0.003 (the experts had learned around the memory). Relying on the table is the design: the 256→180 expert carve removed knowledge, and the memory is its replacement — so the table must always be served whole.

agentfix2 gave back about half of lw5's memory gain in 300 steps, while cross-entropy with the memory on stayed flat or improved (chat 1.2027, general 2.1271). The dependence term described below was already satisfied on every corpus row (gap about +6 nats against a margin of 1.0), so nothing held the table in place while the body kept training. The next run sets the margin above the current gap. The gain is still well above lw2's and every earlier Whittle's.

The jump at lw5 came from one constant. The training term that rewards routing knowledge through the memory compares cross-entropy with the memory off against on, and only applies when the difference falls below a margin. That margin was 0.3 nats while the actual difference ran 1.9 to 5.4, so the term had been satisfied on essentially every row and had stopped doing anything after the first few hundred steps. Raising it to 1.0 keeps it acting on the weakest tenth of rows for the whole run.

The memory's output is ~0.28× the residual norm at layer 2; the difference from earlier versions is that the layers above now use it. Cross-entropy with the memory did not degrade as the gain grew: on held code it is 1.2511 at lw5 against 1.257 at lw2, so the body is leaning on the table rather than being hollowed out.

Not re-measured on agentfix2: teacher parity, the maths probe, the served stop battery, GSM8K and the long-context gate below are lw5's numbers.

Teacher parity (logit-lens agreement of the student's readout with the teacher's, 400 unseen 512-token rows, 3,840 positions per pair):

Maths probe — 60 MATH-train problems (levels 2–4, none in any training set), thinking on, serving sampler, 6,144-token cap, one sample each: tbl1 46/60, lw2 44/60, lw5 43/60 (L2 19, L3 14, L4 10; 6 at the cap). v4.4: 48/60, v4.3: 43/60. The spread across these checkpoints is a handful of problems on a 60-problem probe and should be read as such.

Held-out CE with the memory on: tbl1 chat 1.2098 / general 2.1141; lw2 1.2131 / 2.1304; lw5 1.2055 / 2.1299; agentfix2 1.2027 / 2.1271 (v4.4: 1.2332 / 2.1218).

Served as the Q8_0 GGUF on stock llama.cpp (3× RTX 3060, memory in RAM), serving sampler, thinking on:

Read that long-context row carefully, because we nearly published a wrong number. The gate asks six questions about a real pull request with the diff plus growing amounts of the repository as context, and demands a bare JSON object. It scores all-or-nothing per level: one unquoted value and six correct answers score zero. A single run of it is close to a coin flip — the same checkpoint scored 30/36 and 17/36 on consecutive runs of identical prompts. Across five seeds, reading is indistinguishable between the two checkpoints (4.90 vs 4.93 of 6 when the output parses); what differs is how often the output is valid JSON. Earlier versions of this card quoted a single lucky run; these are means with their ranges. If you need structured output from this model, constrain it with a JSON schema at serve time rather than trusting it to punctuate.

The production Qwen3.6-35B-A3B reviewer we run scores 3/3/2/4/3/3 on the single-sample version of the same gate. Replies for every probe are in eval/.

Not measured on this model: the 200-question GSM8K runs on the Whittle-Next card.

How it was made

1. 1.Memory transfer. Qwen3.8-Flash-Next's 320 M-row table (16 heads × 20 M × 160, fp8) was read shard by shard; for every bigram/trigram in a 172 M-token counting corpus (code, the teacher traces, chat), the exact Qwen row-set (4 kept heads × 160) was written into our 5× larger hash geometry. Bigram buckets end up holding single n-grams on average (mean row norm 1.05× a raw Qwen row-set); trigram buckets still average ~15 colliding n-grams. Unvisited buckets are exact zeros. Hash contract unchanged from Whittle-Next (rows-per-head ×5).
2. 2.Table-first warm-up (30 min): only the memory's projections and rows learned.
3. 3.Joint training with a dependence loss (3.3 h on one 96 GB Blackwell). On a quarter of the corpus rows the trainer runs the same row a second time with the memory zeroed and penalises relu(0.3 − (CE_off − CE_on)): the model is punished whenever it is just as good without the memory. It cannot start from nothing — while the memory carries no information the two forwards have identical gradients — but once anything predictive flows it rewards routing knowledge through the table. The gap went +0.001 → +1.69 nats over the run, with the margin met on 97 % of memory rows at the end. (At this margin the term goes quiet once the gap clears 0.3; see step 5.) Alongside: forward-KL distillation on 1,840 complete Qwen3.8-27B thinking traces (top-128 per position) and a gentle layer-wise steer (student layers 35/39 toward teacher layers 59/63, weight 0.15 on a quarter of steps). → tbl1.
4. 4.lw2: 3,624 more steps, the layer-wise steer at weight 0.2 on half the steps, and PoSE (2k-token rows placed at random position offsets up to 131k, so the rotary positions the model meets at 60–75k context are trained, not extrapolated).
5. 5.lw5 (the root): 1,643 more steps with three changes. The training mix moved to three quarters teacher reasoning traces. The layer-wise steer dropped to weight 0.1 on a quarter of steps and was pointed at a new teacher-layer cache built from maths reasoning rather than general text, because the old cache was the reason the steer cost arithmetic. And the dependence margin went from 0.3 to 1.0, which is what made the memory load-bearing in earnest.
6. 6.agentfix2 (the root): 300 more steps from lw5 for the tool-use loop. lw5 was sampled on 235 synthetic agent contexts (Claude Code-style tools and working directories) and only the replies a grader accepted were kept; the 212 that fit whole in a 2,048-token window (so the task and the working directory are always in view) were trained with cross-entropy on the reply tokens only, on 20 % of steps. A first attempt that also trained on the prompts — random paths, hashes and IDs — taught the model to discount the memory within 200 steps and was stopped.

Run it

llama-server -m Whittle-Qwen-3.8-35B-A3B-Q8_0.gguf -ngl 99 -c 16384 --jinja -fa on -ot per_layer_token_embd=CPU

• —-ot per_layer_token_embd=CPU keeps the 10 B memory (~10.5 GB at Q8) in system RAM: it is read one row per token per head, so the GPU footprint is that of a 27 B-class model and generation speed is that of a 3 B model.
• —Sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05 — sample, do not decode greedily.
• —Thinking: "chat_template_kwargs": {"enable_thinking": true}; give it 4k+ tokens for code. --reasoning-format deepseek separates the block.
• —Architecture qwen4exp; if your build reports an unknown architecture, update llama.cpp.
• —Tool use / agents: keep each assistant turn's reasoning_content in the history for the rest of the tool episode. With it, even lw5 did not loop in our probes. Claude Code works against llama-server's Anthropic endpoint (/v1/messages) with the chat template in this repo and in the GGUFs.

Caveats, measured

• —agentfix2's memory gain is about half of lw5's (see Measured); bf16-lw5/ is the checkpoint with the strongest memory.
• —A report of Claude Code writing to a corrupted working-directory path (/ written as -) did not reproduce in our probes, on Q80 or on the reporter's exact Q5K_M file; if you see it, please open a discussion with the transcript.
• —The memory has learned code first: its gain there is nearly three times its gain on general text and chat, and seven times its gain on science.
• —Maths sits at the level of the 27B-A3B line (43–48/60 on our probe) across every checkpoint in this repo; bf16-tbl1/ is the strongest of them by three problems.
• —Science is the one held set that got worse as the memory got stronger: cross-entropy with the memory on went from 0.756 at lw2 to 0.796 at lw5, the price of a maths-heavy mix.
• —Because the body now depends on the memory, the table must be served whole — this release ships every row (no norm pruning). A GGUF that drops or re-hashes the table will behave like v4.4 minus its knowledge.
• —Structured output: under the serving sampler roughly one reply in three that must be a bare JSON object writes a string value without quotes (correct content, invalid JSON). Until the next distillation round includes structured-output traces, constrain it at serve time: pass response_format with a json_schema (llama.cpp grammar-constrains the decode, so invalid JSON cannot be emitted).
• —Thinking length is unbounded by default: a whimsical prompt can spend 2k+ tokens deliberating. Give max_tokens headroom or cap the reasoning with your server's reasoning-budget option.
• —Teacher voice: long graded replies pull toward the 27B's planning register. Greedy decoding loops on this family; use the sampler above.

Provenance

David Aylward (logic65) & Claude (Anthropic). Parent: logic65/Whittle-Next-27B-A3B (its card carries the full v1–v4.4 lineage back to Qwen3.6-35B-A3B). Teacher: Qwen/Qwen3.8-27B. Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0. No benchmark test sets were trained on; the maths probe problems come from the MATH train split and were excluded from every training corpus.