Modelpublic

Baekpica/Inkling-Small-Mixed-Quant-GGUF

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

1likes6.9kdownloads

Model Card

Inkling-Small — Mixed Quant GGUF

Two independently audited mixed-precision variants of thinkingmachines/Inkling-Small, source revision 8cc5877b44d343f88b92086aa1fb72897950f06a. Choose one variant directory to download. Each contains all of its shards, checksums, tokenizer/processor/config assets, model card and validation reports.

ds4-dfm-rs serves MQ85GB text on one DGX Spark. llama.cpp does not implement this source-layout contract. The bench below is cold ds4-bench with MTP off, not an HTTP serving or 64K result. MQ89 was not timed. No K2, dots, or Qwen throughput number is reused here.

MQ85GB text throughput

Measured 2026-09-22 on `ds4-dfm-rs@f3aef7c`, one GB10, nvidia-smi -lgc 300,2200, prompt promessi_sposi.txt, 8192 cold prompt tokens and 64 greedy tokens, three fresh processes. Under load the SM clock stayed at 2190–2197 MHz.

	Prefill tok/s	Decode tok/s
Median	452.58	13.07
Warp rollback	445.01	13.06
Tile samples	453.50 / 452.58 / 452.56	13.06 / 13.12 / 13.07

The warp row is the same binary with DS4_INKLING_NO_LOGIT_TILE=1. The 8192-token frontier (200058 logits) matched that rollback bit-exactly (max_abs=0) on the earlier unlocked run. Default prefill chunk is 2048. DS4_INKLING_PREFILL_CHUNK=1024 and DS4_INKLING_NO_LOGIT_TILE=1 restore the previous paths. Contexts above 8192, including 64K, were not measured. Write-up: `docs/inkling-optimization-2026-09-22.md`.

Variant directory	Bytes	Decimal GB	GiB	Shards
MQ89	95,446,595,200	95.446595	88.891569	6
MQ85GB	85,732,587,168	85.732587	79.844694	6

GB = 1,000,000,000 bytes; GiB = 1,073,741,824 bytes. MQ89 addresses the original 84–89 GiB weight budget. MQ85GB addresses the additional 84–86 GB budget, saving 9.714008 GB for additional runtime headroom. MTP is separate and excluded.

Download one variant

bash

# MQ89
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ89/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ89 && sha256sum -c SHA256SUMS)

# MQ85GB
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ85GB/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ85GB && sha256sum -c SHA256SUMS)

Both retain all 888 main tensors, all 42 decoder layers, 256 experts in every sparse layer, and complete native BF16 vision/audio ingress and attention. general.architecture=inkling, inkling.tensor_layout=source-interleaved-v1, GGUF quantization version 2. Gate/up interleaving and original tensor names/shapes are unchanged. Read INKLING-CONTRACT.md before native work.

Precision differences

Group	MQ89	MQ85GB
Attention, vision/audio	Original BF16	Original BF16
L2 routed experts	BF16	Q8_0
Embedding/head, dense L0–1, shared experts	Q8_0	Q8_0
L3–39 routed gate/up	IQ2_XXS	IQ2_XXS
L3–39 routed down	Q2_K	IQ2_XS
L40 routed gate/up	Q4_K	Q3_K
L40 down and all L41 routed matrices	Q4_K	Q4_K
Router/norm/bias/scale/convolution/relative controls	Native	Native

MQ85GB changes 40 precisions and preserves the other 848 tensor payloads bit-exactly. Its L2 Q8 payloads also match the separately published Q8 reference. All native payloads match original BF16/F32 source bytes. L2 Q8 is the deliberate large capacity tradeoff; this variant does not retain that official NVFP4 BF16 exclusion. Detailed audits and results are in each variant directory.

Calibration and fidelity

Both use the same measured multimodal imatrix: 1,663 exact processed inputs / 1,578,255 valid decoder tokens. The balanced primary pass has importance weight 1.0; a real structured reasoning/tool coverage supplement has weight 0.1. Effective text/code/image/audio shares are 45.215/21.433/23.817/9.535 percent. The public calibration dataset contains the exact inputs and complete source lineage/weighting details.

All L3–39 target experts were observed. L41 expert 123 remained unobserved; its ordinary Q4_K no-imatrix payloads are identical in both variants. No missing importance vector is fabricated. Shared calibration/ contains NPZ/DAT importance and coverage. reproduction/ and provenance/ record code, compiler/backend context and precise pins.

Offline fidelity suite	MQ89 mean KL vs BF16	MQ85GB mean KL vs BF16	MQ89 top-1 agreement	MQ85GB top-1 agreement
short	0.309541	0.382427	0.824	0.784
extended	0.133307	0.146076	0.918	0.920

These are small teacher-forced diagnostics (including real document/speech), not task benchmarks or guarantees of equivalent quality. Individual probes, direct variant comparisons and 32-token-budget generation checks are included. Extended media were excluded from calibration; text rows were excluded from the structured supplement, but overlap with historical primary text pools cannot be fully ruled out. Offline GGUF weights were dequantized to BF16 on B300 GPUs, so those timings/memory figures are not native quantized serving data.

The weight budget leaves room for caches and activations but does not prove that a particular context fits on GB10. Seven global-attention layers alone need about 28 GiB BF16 KV at 1,048,576 tokens, before local state, activations and OS memory. Actual admission and serving measurements remain necessary.

Related artifacts

The earlier MQ89 root-level paths remain available at historical revision f091d3672dc69a1af77cad92b63cdeddc145a039; current main uses variant directories. The MQ89 weight bytes and hashes did not change during this organization. Source model is Apache-2.0; dataset sources retain their respective licenses.