Anjielon/ODINO-397B-v34a-TQ1_0

<p align="center"> <img src="assets/odino_logo.png" alt="ODINO" width="260"/> </p>

<h1 align="center">ODINO 397B v3.4a</h1> <p align="center"><b>Two-plane ternary TQ1_0 · 88.2 GiB · 1.911 bits/weight</b><br/> a 397B-parameter, 512-expert MoE forged on a single desktop machine</p>

<p align="center"> <a href="https://github.com/ggml-org/llama.cpp/pull/27765">llama.cpp Vulkan TQ1_0 kernels (PR #27765)</a> · <a href="https://github.com/Anjielon/fucina">the forge (paper + tools)</a> </p>

The headline

+23.1 points on real coding-and-agentic work, discordant pairs 39–6, exact McNemar p = 5.42e-07, paired-bootstrap CI₉₅ of the difference [+14.7, +31.5] points.

⚠️ The sizes are not identical and we do not round them away: the IQ1_M reference is 3.5% smaller (the ternary file is 3.6% larger), so the gap above comes with a small size handicap against us, not in our favour.

⚠️ How this file was served in the two evaluations (found while preparing the revised paper, by reading the file's tensor table against the engine's default). The file carries second-plane (_exps2) tensors on all 60 layers, and our llama.cpp fork applies a second plane wherever those tensors exist unless ODINO_NO_P2=1 or ODINO_P2_LAYERS=LO-HI restricts it. The public-benchmark table below was run with no environment variable, i.e. with the second plane applied on all 60 layers. The task-suite numbers above were served by our launcher script, which sets `ODINO_NO_P2=1` (no second plane at all); the per-run environment was not recorded, so this is what the launcher sets, not a logged fact. Neither configuration is the tail-only (layers 44-59) recipe of the paper's Section 5, whose evidence rests on engine-switch measurements, not on these tables. The first plane is RTN on layers 0-43 and GPTQ on layers 44-59 (transplanted), per the forge record.

The finding: perplexity cannot tell this pair apart — the public benchmarks can

Nine public benchmark corpora run whole plus a declared 1,000-item PIQA subset, same engine (our llama.cpp fork, Vulkan), same GPU (NVIDIA RTX PRO 6000 Blackwell, 96 GiB), one model per GPU, reasoning off. Accuracy is llama.cpp's per-token-normalised score with a Wilson 95% interval; bold marks the three benchmarks whose intervals are disjoint. Because both files were scored on identical items in identical order, the paper also reports the paired per-item exact McNemar test from the raw logs: at the Bonferroni-corrected level (0.05/10) it separates four benchmarks — the three in bold plus ARC-Easy (207-148 discordant, p = 0.002); Winogrande (247-189, p = 0.006) is nominally significant and just misses the corrected threshold; ARC-Challenge, OpenBookQA, PIQA and TruthfulQA MC1 are not separated at these sample sizes.

Δ = ours − rival; for perplexity lower is better, so the +0.0007 there is nominally in the rival's favour (one-seventieth of its standard error, and a paired per-chunk difference of 0.01 standard errors). No corpus shows a negative difference beyond its interval; the ternary file wins three of them outright (BoolQ by eight points), and the two files are indistinguishable on perplexity — the cheapest and most widely reported of these measurements is the only one that cannot see the difference that the private task suite above puts at +23.1 points.

The earlier home measurements, and a correction

[image]

The figure and the table below are the measurements first published with this model, taken on the AMD desktop APU (gfx1151, Vulkan/RADV); the raw logs are in `logs/`.

⚠️ Correction. This page previously described the HellaSwag row as the "same 2,108-item prefix". That was wrong: llama.cpp's HellaSwag scorer always draws tasks at random without replacement (fixed seed 1), so the 2,108 items are a random subset in random order, and the ODINO run had been launched for 4,000 tasks and stopped at 2,108 (see the selecting ... randomized tasks lines in the logs). Both files were scored on the same 2,108 items, so the comparison was paired — but the sample is superseded by the whole-corpus figures above. The perplexity row was also over-read: 7.5903 vs 7.5438 is inside its own error bars, and on the second GPU the two files come out identical (7.5180 vs 7.5173). The same items on the two GPUs agree within one combined standard error (Winogrande 65.67 → 66.22 and 61.09 → 61.64; perplexity 7.5903 → 7.5180 and 7.5438 → 7.5173). Paired on the same items the picture is finer, and the paper reports it: 14.6% of Winogrande items change outcome between the two GPUs on this file (balanced, McNemar p = 0.66), and the perplexity offset on this file is systematic once paired per chunk (+0.96% at home, t = 3.9 over 72 chunks) — a declared backend difference that changes no comparison, since both files are measured on the same GPU in the table above.

Task results by family

[image]

Reasoning off, two fixed seeds (7, 42), identical server and judge for both files. Frozen data: bench_v2_CONGELATO_20260829-1959.csv, md5 66c0d923c2818bb306b578a5e10cedd0.

Files

hf download Anjielon/ODINO-397B-v34a-TQ1_0 ODINO-397B-v34a.gguf --local-dir .

Running it

The single-plane TQ1_0 Vulkan path is upstream: llama.cpp PR #27765 (merged 2026-09-07). The same shaders have been exercised on AMD gfx1151 (Radeon 8060S, where this model is served daily), on an NVIDIA RTX 4060 and on NVIDIA RTX PRO 6000 Blackwell (the benchmark table above) — no vendor-specific changes. The quantizer itself is bit-exact across CPU and CUDA backends (identical packed payloads, |Δ| = 0).

llama-server -m ODINO-397B-v34a.gguf -ngl 999 --ctx-size 4096   # ~88 GiB weights

On a 128 GB unified-memory APU the model runs fully resident. ⚠️ The second ternary plane (the _exps2 tensors and their application) is not upstream: on stock llama.cpp the file is rejected at load (done_getting_tensors: wrong number of tensors; expected 1278, got 1098 — the 180 unrecognised _exps2 tensors; verified 2026-09-20 on an unmodified upstream checkout). Only the single-plane TQ10 *format* is upstream; this two-plane *file* needs the fork. The forge scripts are in [the forge repository](https://github.com/Anjielon/fucina); the two-plane **loader** is not yet in any public repository — the engine used for every run is released with the revised paper's supplementary material as two diffs with their SHA-256. The fork's git history was not preserved (the binary reports `commit unknown`), so the upstream commit it was branched from was identified afterwards by matching every file the fork does not touch against upstream history: all of them are blob-identical to the tree of `521a64cd01979bb5b1a466152c576a9d809b068d` (2026-08-20), and the fork changes 19 files relative to it (the TQ10 Vulkan kernels and shaders, the two-plane loader in src/, an activation-dump facility, one test). The second diff is against 8fe90e1fbfc065f17a0b233c9df239423cd24a75, the commit that merged PR #27765. To rebuild: check out the base commit, apply the first diff, cmake -B build -DGGML_VULKAN=ON. Runtime switches: ODINO_P2_LAYERS=LO-HI restricts the second plane to a band, ODINO_NO_P2=1 disables it; with neither set it is applied on every layer whose _exps2 tensors exist (all 60 in this file).

Reproducing the numbers on this page

Sources: HellaSwag klosax/hellaswag_text_data (hellaswag_val_full.txt, llama.cpp get-hellaswag.sh); MMLU cais/mmlu config all, test split; ARC allenai/ai2_arc, test splits; TruthfulQA MC1 truthfulqa/truthful_qa config multiple_choice, validation split (817 items, lm-eval's fixed six-shot preamble, up to 13 options per question); PIQA gimmaru/piqa validation — a 1,000-item subset of the official 1,838-item `ybisk/piqa` validation split (verified: every row and label is in the official split, no duplicates, not a prefix; declared as a subset, not a whole corpus); OpenBookQA allenai/openbookqa main, test; BoolQ google/boolq, validation; Winogrande winogrande-debiased-eval.csv (llama.cpp get-winogrande.sh); WikiText-2 raw wiki.test.raw (llama.cpp get-wikitext-2.sh). The multiple-choice sets are converted to llama.cpp's binary --multiple-choice format by converti_multiple_choice.py (released with the paper's supplementary material, with the SHA-256 of every converted file), zero-shot except TruthfulQA MC1, with these prompt templates: MMLU The following are multiple choice questions (with answers) about {subject}.\n\n{question}\nA. …\nB. …\nC. …\nD. …\nAnswer: scored over the letters A/B/C/D; ARC and PIQA Question: {q}\nAnswer: over the option texts; OpenBookQA the bare stem; BoolQ {passage}\nQuestion: {q}?\nAnswer: over no/yes; TruthfulQA MC1 lm-eval's fixed six-example QA preamble (tasks/truthfulqa/truthfulqa_mc1.yaml, copied verbatim), prepended to every question, over the mc1_targets option strings. The template decides the number (an earlier BoolQ template without the scaffold put this file below chance).

# perplexity (the metric that cannot tell this pair apart)
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 4096 -b 2048 -f wiki.test.raw

# HellaSwag: every task of the file (random order, seed 1, without replacement)
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 4096 \
    --hellaswag --hellaswag-tasks 100000 -f hellaswag_val_full.txt

# MMLU / ARC / PIQA / OpenBookQA / BoolQ: every task, in order.
# -np 8 is REQUIRED: ARC has 5-option items and with the default -np 4
# llama-perplexity stops there while reporting a result.
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 8192 -np 8 \
    --multiple-choice --multiple-choice-tasks 100000 -f FILE.bin

# Winogrande: all 1,267 items
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 4096 \
    --winogrande --winogrande-tasks 100000 -f winogrande-debiased-eval.csv

# TruthfulQA MC1: up to 13 options per question, six-shot preamble baked
# into FILE.bin; batch reduced to fit this file's compute buffers
llama-perplexity -m ODINO-397B-v34a.gguf -ngl 999 -c 3328 -np 13 -b 256 -ub 256 \
    --multiple-choice --multiple-choice-tasks 100000 -f FILE.bin

The raw output of the original home runs (perplexity, HellaSwag 2,108 random, Winogrande), for this file and for the IQ1_M baseline, is in `logs/`. Numbers are llama.cpp's per-token-normalised scores: comparable between these two files, not with lm-eval leaderboards.

Honesty of the measurement

• —The task suite is private: aggregates are published, fixtures are not. Families built on the author's own home automation are excluded from every number on this page.
• —A number without its raw log does not enter this page. The raw perplexity / HellaSwag / Winogrande logs of the home runs for both files are in `logs/` in this repository; the whole-corpus runs were scored chunk by chunk (HellaSwag in 21 chunks of 500 tasks) and every raw log, including discarded ones, is kept and released with the revised paper's supplementary material.
• —Generative benchmarks are not on this page. Everything here is log-likelihood scoring of given choices; GSM8K, HumanEval and the like need a harness against a server and are deferred.
• —Runs whose harness logged a transport fault (5xx, dropped connection, malformed response) are never scored as model failures: the run is discarded and re-executed.
• —Four figures from an earlier night had been transcribed by hand while their logs sat in a temporary folder the system cleaned. They were re-measured from scratch rather than copied from memory — the transcribed Winogrande said 64.80; the true value is 65.67.

What is NOT yet demonstrated

The forge has one measured but not-yet-spendable lever: a Hadamard rotation (incoherence processing in the sense of QuIP#/QuaRot/SpinQuant) over the full 256-weight TQ1_0 block that removes ~5% of reconstruction error at the same bit budget on this model's tensors (0/40 tensors worse), waiting on the decode-side kernel that undoes it. It is stated here so that the current file is not mistaken for the ceiling.

Citation

The accompanying paper is under review; its title will be added here when the review concludes.

@misc{odino2026,
  title  = {ODINO 397B v3.4a: a two-plane ternary TQ1\_0 quantization of
            Ornith-1.5-397B (paper under review)},
  author = {Anjielon},
  year   = {2026},
  note   = {Model card and forge: github.com/Anjielon/fucina}
}