xautonomics/Inkling-Small-UD-Q4_K_M.LAYR.GGUF

Inkling-Small-UD-Q4KM.LAYR.GGUF

A per-layer GGUF library of Unsloth's unsloth/Inkling-Small-GGUF (UD-Q4_K_M). These are the same quantized weights, sliced once into one file per transformer block plus the shared parts, so any contiguous layer window [A, B) can be assembled at load time. It is built for dreamcatcher, xAutonomics' fork of ik_llama.cpp, which serves a library either in a single process or split across hosts.

Support status

This is the text-only Inkling port tracked in dreamcatcher issue #43. What we have and have not validated, as of publication:

Validated

• —Cross-lineage envelope on the real 152 GB model (the monolithic UD-Q4_K_M source of this library, measured with KL-divergence against a banded oracle produced by the upstream llama.cpp draft build, ggml-org/llama.cpp#25731): the D2 masked attention path reaches 83.02 % top-1 agreement and the D3 banded attention path 83.87 %, both inside the pre-registered 17.08 % working band (the oracle build's own masked-vs-banded self-disagreement).
• —Remote routed experts: the expert-server path is byte-exact against in-process experts.
• —Stage ring: a head/tail llama-stage-runner ring runs token-identical to the monolith on the synthetic Inkling fixture that carries this model's scalar hyperparameters.

Not yet validated / in progress

• —Greedy token parity against the upstream llama.cpp draft (#25731) build on this model — tracked in #43.
• —This library has been hash-verified against its source tensors (every file and every tensor, see manifest.json), but the numbers above were measured on the monolithic source GGUF; the library-assembled load of this model is not yet on record.

Unvalidated

• —Multimodal (audio/vision): the port is text-only. The mmproj-* projector files of the source repository are not included and are not supported.
• —The measurements above were taken with the fleet's CUDA build; other backends (Vulkan, SYCL, CPU-only) are not measured for this architecture.

Only model files from huggingface.co/xautonomics are supported by dreamcatcher. Other GGUFs, including libraries you slice yourself, may load but are unsupported. Do not rename the files: the loader expects blk-NNNNN.gguf / parts-*.gguf / manifest.json exactly.

Layout

blk-00000.gguf … blk-00041.gguf   one transformer block each (blocks 0 and 1 are the dense blocks)
parts-embd.gguf                          token embedding (every stage loads it)
parts-output.gguf                        output norm + head (every stage loads it)
parts-other.gguf                         remaining non-block tensors, here `token_embd_norm.weight` (every stage loads it)
manifest.json                            file list, sizes, per-file + per-tensor hashes, source provenance

A stage that serves layers [A, B) needs blk-A … blk-(B-1) plus the parts-* files and manifest.json. Each file keeps the full source metadata (tokenizer included), and carries its own contribution to inkling.block_count / inkling.dense_block_count, which the loader sums over the assembled window.

Use

Single process (the whole model on one host):

llama-server --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF

Multi-host ring (stages exchange hidden states over TCP; each host needs low-latency, direct node-to-node links — see the dreamcatcher README):

llama-stage-runner --role tail --listen 8081 --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF --layers 21,42
llama-stage-runner --role head --connect <tail-host>:8081 --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF --layers 0,21

Plan a split for your hosts (weights-only screen; the proof is a survived prefill), and verify the download before launching:

python3 -m layer_distribution.plan /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF/manifest.json --node a:16GiB:200GiB:8080 --node b:24GiB:200GiB:8081
python3 -c "from layer_distribution import verify; r = verify('/models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF', '/models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF/manifest.json'); print(r.passed, r.hash_verified)"

Provenance and license

Weights are byte-for-byte the tensors of unsloth/Inkling-Small-GGUF at revision 1a19ef82 (Unsloth's dynamic quantization of thinkingmachines/Inkling-Small). The five source shards were checked against the Hub's LFS SHA-256 digests before slicing; their names and sizes are recorded in manifest.json (source.shards[], source.content_hash). License follows the base model: Apache-2.0, see LICENSE. Credit to Thinking Machines Lab for the model and to Unsloth for the quantization.

About

Published by xAutonomics for dreamcatcher.