EldanRing/Winnow-12B

Winnow-12B — GGUF

Local Jev-style decisions, chat, and vision. Q8 tested with 64K context and vision on a 16 GB RTX 5070 Ti.

Winnow-12B fine-tunes Gemma 4 12B IT for typed decisions. Its llama.cpp-based inference server provides /v1/systemone and ordinary /v1/chat/completions from the same loaded model.

• —Typed decisions: ask noul, choice, and score questions against shared state.
• —Shared computation: prefill the state once, fork question branches, and read answer-token logits without generating answer text.
• —Chat and vision: regular chat, streaming, and image inputs through the same server.
• —Two GGUF model downloads: BF16 (16-bit floating point) and Q8_0 (8-bit quantized), both ready for llama.cpp. Neither needs conversion or a separate LoRA adapter.

Inference code · Quickstart · Full benchmark report · Artifact manifest

GGUF downloads

Choose one model file. Both versions run directly in the Winnow llama.cpp-based server. The optional vision projector is listed separately below.

BF16 weights alone exceed 16 GB VRAM; the 5070 Ti full-offload measurements below apply to Q8_0. Text-only use needs just the chosen model GGUF. Vision needs that model plus mmproj-Winnow-12B.gguf.

Optional vision projector

**Download mmproj-Winnow-12B.gguf** — 175 MB / 0.163 GiB.

This is the F16 vision projector, not another model or quantization choice. Use it alongside either BF16 or Q8_0 for image inputs; skip it for text-only use. The same projector works with both model downloads above.

This repository distributes GGUF model weights only. No safetensors shards or conversion step are required. See Quickstart for exact download and launch commands, and SHA256SUMS for file checksums.

Decision quality

[image]

Both Winnow variants below were evaluated in the same RTX PRO 5000 Blackwell campaign. Jev was evaluated through OpenRouter on the same frozen inputs and scoring rules. Both BF16 and Q8 were served as GGUF models. The released GGUF files are listed above; evaluation hashes are in the benchmark report.

Winnow Q8 matches Jev on this JevBench public subset: 198 of 231 correct. This is a result on that subset, not a claim of universal parity. JevBench here means public-subset accuracy, not the official composite leaderboard score. The complete report includes competitor comparisons, all measured suites, calibration, hardware and evaluation scope.

RTX 5070 Ti: 64K context with vision

[image]

The released Q8 model and matching projector were tested with full GPU offload, Q8 KV cache, four decision branches, one chat slot, and exclusive memory scheduling.

These are separate capacity and timing probes, not a single simultaneous workload. The host had 96 GB system RAM; the observed PSS is not a minimum installed-RAM recommendation. Full hardware, repetition counts, and timing definitions are in the report.

64K includes prompt formatting, image positions, questions, and generated output. In the tested exclusive profile, chat and decision requests share the weights but take turns using their KV contexts; switching can evict a cached prefix.

GGUF packaging

Each model GGUF contains its language weights, tokenizer, and chat template. BF16 and Q8_0 are exports of the same merged fine-tune; they are alternative model files, not parts to combine. The F16 projector supplies vision support. The remaining root configuration/tokenizer files are reference assets; llama.cpp loads the GGUF directly. No separate LoRA adapter is needed.

Training

Winnow is a LoRA fine-tune, released after merging the learned update into the base model. It is not a full-parameter training run.

The training dataset is private and is not released. It contains curated typed-decision examples: synthetic scenarios, teacher-supervised examples, and labeled semantic tasks. Task coverage includes routing, policy and rule application, evidence selection, workflow decisions, ordinal judgments, entailment, paraphrase, and answerability. Contrastive examples vary facts that should change an answer. An initial decision fine-tune was followed by a 26,000-example refinement mixture that included replay of earlier examples. This release was selected after 19,200 refinement examples had been processed; these are training presentations, not a claim of that many unique new examples. Refinement used BF16 LoRA, an effective batch size of 8, and an 8,192-token training cap. The objective combined gold-label cross-entropy with teacher-distribution cross-entropy only when the teacher agreed with the gold label.

Training and validation were split; the public benchmark report documents the separate evaluation scope and any known development exposure. No claim is made about excluding public benchmarks from the base model's pretraining data.

Probabilities and scope

The decision server normalizes logits over the supplied answer options. Its entropy-based confidence summarizes concentration within that distribution; it is not a guaranteed probability of correctness. Reported default decision temperature is 1.0, without a separately fitted calibration map.

Chat and image input are functional in the release runtime. A complete paired general-chat quality comparison against the unfine-tuned base was not completed; this release does not claim identical chat quality. Near-full-context retrieval checks establish capacity and operation, not general 64K reasoning quality. Audio/video capability is not evaluated by this release.

Credits and license

Winnow-12B is an independent fine-tune by EldanRing of Google DeepMind's Gemma 4 12B IT, released under Apache 2.0. See LICENSE and NOTICE.

The separate inference repository builds on llama.cpp by Georgi Gerganov and contributors and preserves its MIT license. Jev-style refers to the typed-decision interface; Winnow is not affiliated with or endorsed by TypeSafe, Google, or llama.cpp.