Modelpublic

0xSero/Inkling-Small-EXL3-2.0bpw

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes378downloads

Model Card

Inkling-Small-EXL3-2.0bpw

Calibrated EXL3 trellis quantization of the routed MoE experts in thinkingmachines/Inkling-Small, targeting 2.0 bits per routed-expert weight.

Status: The assembled weight archive has been uploaded. Consult EXL3_MANIFEST.json for the current structural and runtime validation state.

Runtime validation: Text generation, multimodal generation, and MTP validation are pending. The uploaded archive and its packed tensors have structural validation only.

What is quantized

Component	Storage
Routed MoE experts, layers 2–41	EXL3/MCG trellis, 2.0 bpw target
Dense MLP layers 0–1	Source BF16
Shared experts and routers	Source BF16/FP32
Attention, relative-position, and short-convolution tensors	Source precision
Embeddings, norms, and LM head	Source precision
Vision/audio components	Source precision
Eight MTP layers	Source precision

All forty routed MoE layers use integer EXL3 K=2 trellis weights.

—Calibration: 1,048,576 naturally routed tokens selected with seeded, no-repeat axis water-filling across general, legal, code/agentic, and reasoning/termination data
—Maximum calibration sequence/sample span: 4,096 tokens
—Routing: Inkling's natural top-6 routed-expert assignments
—Source revision: b2d4f225a02032c5d154bff748ab5a00c5ca26e4
—Achieved routed-trellis rate: 2.000000 bpw
—Assembled repository payload: 75.75 GiB
—Per-layer allocation, tensor inventory, sizes, and validation state: EXL3_MANIFEST.json

Compatibility and how to use it

Download the repository with:

bash

hf download 0xSero/Inkling-Small-EXL3-2.0bpw \
  --local-dir Inkling-Small-EXL3-2.0bpw

This repository is not a drop-in Transformers checkpoint. The routed experts use EXL3 trellis tensors while the rest of Inkling remains in source precision. It requires an Inkling-aware EXL3 loader/runtime that understands the tensor layout described by quantization_config.json and EXL3_MANIFEST.json.

Stock ExLlamaV3 v1.2.1 does not yet include an InklingForConditionalGeneration architecture adapter. The upstream BF16 Inkling model has vLLM and SGLang recipes, but those recipes do not by themselves add support for this experts-only EXL3 layout. Do not infer text, image, audio, or MTP runtime support from a successful download or structural assembly alone.

All EXL3 variants

Target	Repository
2.0 bpw	0xSero/Inkling-Small-EXL3-2.0bpw
2.5 bpw	0xSero/Inkling-Small-EXL3-2.5bpw
3.0 bpw	0xSero/Inkling-Small-EXL3-3.0bpw
3.5 bpw	0xSero/Inkling-Small-EXL3-3.5bpw
4.0 bpw	0xSero/Inkling-Small-EXL3-4.0bpw
4.5 bpw	0xSero/Inkling-Small-EXL3-4.5bpw
5.0 bpw	0xSero/Inkling-Small-EXL3-5.0bpw

Method

The source model is loaded once for calibration. Hidden states and natural expert assignments are captured for all forty routed layers. Each expert's gate, up, and down projections are calibrated, Hadamard-transformed, and encoded as EXL3/MCG trellis weights. The full sweep checks finite Hessians and scales, exact trellis byte counts, safetensor key counts, and per-file checksums. Before the sweep, a bounded H200 proof on a real Inkling expert also passed trellis pack/unpack/repack equality and finite reconstruction. That bounded kernel proof is not a full-model generation test. Integer-K caches are reused to assemble the seven public variants without repeating the full model calibration.

The target bpw applies to routed-expert trellis weights. The complete repository is larger than a whole-model quantization at the same nominal bpw because attention, shared experts, multimodal components, the LM head, and MTP remain in source precision.

Credits

—Thinking Machines Lab / Inkling team for releasing Inkling-Small.
—TurboDerp for ExLlamaV3 and the EXL3 format.
—JarvisLabs for providing the 8× NVIDIA H200 compute used for this quantization sweep.

This is an independent community quantization and is not an official release from Thinking Machines Lab, TurboDerp, or JarvisLabs.

License and use

This derivative follows the upstream Apache 2.0 license and the upstream acceptable-use policy. Review the base model card for intended uses, limitations, and safety information.