sabrewing-engine/Inkling-Small-colibri-int4

Inkling-Small — colibri int4

Thinking Machines Inkling-Small converted to the colibri/sabrewing snapshot format: int4 routed experts with bf16 residents. 131 GiB, down from 495 GiB in bf16.

This is not a transformers checkpoint. The tensors are packed for sabrewing, a dependency-free C engine for MoE inference. AutoModel.from_pretrained will not load it. If you want the original weights, use the base model.

Why this exists

Inkling-Small is ~266B total parameters. In bf16 that is 495 GiB — more than fits in a workstation. At int4 the routed experts (which are ~95% of the weights) drop to 131 GiB total, which fits in RAM on a 187 GB machine, so decode never touches the NVMe expert-streaming path that the 975B Inkling depends on.

Quickstart

hf download sabrewing-engine/Inkling-Small-colibri-int4 --local-dir ~/Models/inkling_small_i4

git clone git@github.com:Schneewolf-Labs/sabrewing.git && cd sabrewing/c
make inkling ARCH=native                  # CPU; add CUDA=1 for the GPU resident tier

SNAP=~/Models/inkling_small_i4 ./inkling -p "The capital of France is" -n 64

Put the snapshot on NVMe, not spinning disk. The engine reads expert weights from the snapshot on cache misses; on an HDD that dominates everything else.

What is in the container

Quantization is per-row symmetric, low nibble = even column, np.rint matching the C engine's lrintf. Identical math to the other colibri int4 containers.

The base model is multimodal (image-text-to-text); this conversion is text-only. The vision and audio towers are not included, and neither is the MTP draft head.

Architecture

Inkling-Small is a straight scale-down of Inkling — same MoE structure, smaller dimensions. No engine changes were needed to run it.

Measured performance

Ryzen 9 7900 (12c/24t), 187 GB DDR5, snapshot on NVMe, CPU build, ARCH=native. 24-token greedy decode, single runs.

At 100% cache hit there is zero disk I/O (fill 0.0s), so the warm number is pure compute. The novel-prompt number is dominated by loading the ~18% of experts the pins missed.

If you get ~0.3 tok/s, check your thread count

OpenMP defaults its team size to nproc. On an SMT machine that runs two threads per physical core, both contending for the same vector units, and the int4 kernels fall off a cliff — measured ~10x on this model and ~28x on Laguna:

sabrewing sets one thread per physical core automatically as of PR #10. On older builds, set OMP_NUM_THREADS to your physical core count by hand.

Cache warming

The engine counts expert selections per (layer, expert) and writes .coli_usage into the snapshot directory, pinning the top experts on the next start. Counts accumulate across runs, so warm it on a diverse prompt set — pins trained on one prompt overfit badly. That file is deliberately not shipped here; it is generated locally from your own workload.

SNAP=~/Models/inkling_small_i4 ./inkling -f warmup_prompts.txt -n 32

Reproducing the conversion

hf download thinkingmachines/Inkling-Small --local-dir <bf16-dir>
python3 c/tools/convert_inkling_int4.py --indir <bf16-dir> --outdir <out-dir>

--watch converts shards while the download is still running. That is a win when the source is on NVMe; when source and output share one spinning disk it is a ~3x pessimization (measured 197 MB/s serialized vs 60 MB/s overlapped) because the head thrashes between the download's writes and the converter's reads. Serialize on HDD.

Caveats

• —Not validated token-exact. sabrewing's usual bar is bit-for-bit agreement with transformers via a tiny-model oracle. The oracle builder needs transformers >= 5.7, which was not available on the conversion machine, so this container has not been checked against the reference implementation. Kernel correctness was verified separately (make kernel-check passes: int4/int8 kernels against a double-precision dequant reference), and generation is coherent — but that is weaker than the usual guarantee.
• —CPU only, so far. The CUDA resident tier was not exercised; the test machine's GPU was occupied. Expect it to help — at 100% cache hit this model is compute-bound, which is what a GPU fixes.
• —Text-only; no vision, audio, or MTP.

License

Apache 2.0, inherited from the base model.