sh111111111111111/Qwen3.6-27B-DF11

Qwen3.6-27B-DF11 — lossless DFloat11 compression

Lossless DFloat11 compression of `Qwen/Qwen3.6-27B`. The weight shards are ~32% smaller and reconstruct the original bf16 weights bit-for-bit — so the model is mathematically identical and accuracy is unchanged by construction (not independently re-benchmarked here). In clean runs, greedy generation matched bf16 token-for-token; see Provenance & correctness for the runtime caveats.

This repo contains: the DFloat11 weight shards, the raw bf16 MTP head for speculative decoding, the vision tower, and the image + video preprocessors. It is a drop-in lossless replacement for the complete multimodal model.

For a smaller download for standard (non-speculative) decoding? Skip the MTP head with --exclude "mtp_head/*" (see Download below).

TL;DR

• —Lossless (weights). DFloat11 reconstructs the exact bf16 bits — compression, not quantization — so accuracy is unchanged by construction. Runtime check: greedy tokens matched bf16 on 8/8 reference prompts (48-token continuation, up to batch-8) in a clean run; runtime logits are not guaranteed bit-identical on every prompt (see Provenance).
• —~32% smaller. 55.6 GB of bf16 weights → ~37.3 GB of DFloat11 weight shards.
• —Memory-first, not speed. DFloat11 is a lossless memory optimization; decode pays a fixed per-forward decompression cost — about ~2–2.5× slower single-stream (measured ~2.48× for a 32-token greedy run); the per-forward cost is expected to amortize with batch, though we did not observe that on this box. On this GB10 / aarch64 box the serving paths were additionally slow or finicky (no aarch64 vllm-df11 wheel; a per-forward custom engine that runs but is far slower; transformers==5.10.1 produced garbage) — those are kernel/library issues local to this hardware, not inherent to DFloat11. See Throughput & memory tradeoff.
• —Lossless + MTP compose. Measured across BF16, BF16+MTP, DFloat11 and DFloat11+MTP in one harness: all four produce byte-identical greedy output, and DFloat11 frees ~14.5 GB of resident memory. You can stack lossless compression and speculative decoding and stay bit-exact.

What is DFloat11?

DFloat11 (Dynamic-Length Float) losslessly entropy-codes the BFloat16 exponent field with Huffman coding, leaving the sign and mantissa untouched. Weights shrink to 11 bits on average (~30%) with no numerical change, and are decompressed on the fly on the GPU just before each matmul, then discarded. The model you run is mathematically the original bf16 model, not a lower-precision approximation. See the paper (arXiv:2504.11651, NeurIPS 2025) and the DFloat11 org on the Hub.

Variants

The DFloat11 weight shards — which include the vision-tower weights — account for 37.3 GB, the 32% saving versus the 55.6 GB bf16 weights. The repo total (37.9 GB) additionally carries the raw bf16 MTP head (0.9 GB, back-derived from the repo totals) and the image/video preprocessor config assets.

The YAML uses base_model_relation: quantized because the Hub has no lossless-compressed relation; DFloat11 is lossless compression, not quantization.

Download (full vs. lean — one repo, two commands)

You do not need a separate repo for the text/standard-decoding case — just skip the MTP head at download time:

# Full build (includes the bf16 MTP head for speculative decoding):
hf download sh111111111111111/Qwen3.6-27B-DF11 --local-dir ./Qwen3.6-27B-DF11

# Lean build (skip the MTP head — standard decoding only):
hf download sh111111111111111/Qwen3.6-27B-DF11 --local-dir ./Qwen3.6-27B-DF11 \
  --exclude "mtp_head/*"

The MTP head is only ~0.9 GB, so the smaller download saves little — keep it unless you specifically want the minimal footprint. The compressed weight shards (including the vision tower) are identical either way; the lean download simply omits the speculative-decoding head.

Running (recommended: transformers)

pip install -U "dfloat11[cuda12]" transformers accelerate
# CUDA 11? use:  pip install -U "dfloat11[cuda11]"

Two runtime caveats we hit on this hardware (NVIDIA GB10 / aarch64). (1) Load in a fresh process. Loading the original bf16 model first and then DFloat11 in the same process produced invalid/NaN logits in a strict direct-forward test; clean DFloat11-only loading was finite. (2) Transformers version sensitivity — we saw garbage output on `transformers==5.10.1`; if generations look wrong, pin a DFloat11-supported transformers release. Both are decode-integration issues local to this box's library/kernel situation, not defects in the shards — the weights themselves are lossless.

import torch
from accelerate import init_empty_weights
from transformers import AutoConfig, AutoModelForImageTextToText, AutoTokenizer
from dfloat11 import DFloat11Model

base_model  = "Qwen/Qwen3.6-27B"
dfloat_repo = "sh111111111111111/Qwen3.6-27B-DF11"

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
config    = AutoConfig.from_pretrained(base_model, trust_remote_code=True)

# Build an EMPTY bf16 model from the base architecture, then load the compressed weights into it.
# (Do this in a fresh process — see the caveat above.)
with init_empty_weights():
    empty = AutoModelForImageTextToText.from_config(
        config, dtype=torch.bfloat16, trust_remote_code=True)
empty = empty.to_empty(device="cpu")

model = DFloat11Model.from_pretrained(
    dfloat11_model_name_or_path=dfloat_repo,
    bfloat16_model=empty,        # the empty base to decode into
    device="cuda:0",
    device_map="auto",           # shards across multiple GPUs via Accelerate if present
    cpu_offload=False,
)
model.eval()

messages = [{"role": "user", "content": "Explain lossless weight compression in one paragraph."}]
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda:0")
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=256, do_sample=False)  # greedy
print(tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Multimodal (image / video). The vision-tower weights are inside the DFloat11 shards, and the image + video preprocessor assets are bundled in this repo. Load the bundled processor with AutoProcessor.from_pretrained(dfloat_repo, trust_remote_code=True) and follow the multimodal usage from the base `Qwen/Qwen3.6-27B` model card — the DFloat11 build is a drop-in for the weights, so the input/output API is unchanged.

Throughput & memory tradeoff

Measured on a single NVIDIA GB10 (unified LPDDR5X, ~273–301 GB/s), greedy, with a small number of reps and no error bars — treat every absolute number here as indicative of one box, one run, not a benchmarked mean. Single-stream decode is memory-bandwidth-bound: reading the ~55.6 GB of bf16 weights per token caps decode at ~5 tok/s regardless of compute, so ratios travel better than absolute rates.

bf16 via vLLM (native engine, CUDA graphs), greedy. Note — these are the uncompressed bf16 base model's numbers. They require the full bf16 weights served through vLLM, not the DFloat11 shards in this repo (there is no aarch64 vllm-df11 kernel). They show the MTP lever on the base; for the DFloat11 weights' own rates see DFloat11 (compressed) serving and the four-config table below.

→ MTP speculative decoding is the single-stream speed lever: ~1.65× (with num_speculative_tokens=1 it verifies up to one extra token per pass, partly amortizing the bandwidth wall). MTP requires this full repo's head.

[image]

DFloat11 (compressed) serving. DFloat11 is a lossless memory optimization, not a speed one, and on GB10 there is a real trade-off between the two paths we measured. Faster but fragile: the transformers + DFloat11 CUDA-kernel path ran a 32-token greedy decode ~2.48× slower than bf16 (≈0.41× the speed) — but this is the path with the runtime caveats below (garbage on 5.10.1, NaN in mixed-process loading). Slower but verified: the custom vLLM eager engine (per-forward Python decode) is much slower — tens of × (see the four-config table) — but is the path that produced the 8/8 bit-exact result. The per-forward cost should amortize with batch in principle; we did not observe that on this box. Two serving paths exist:

• —`vllm-df11` plugin (pip install vllm-df11) — keeps weights compressed in vLLM. Prebuilt wheels are x86_64-only as of June 2026 (no aarch64 wheel), so it does not install on Grace/ARM boxes like GB10.
• —`DFloat11Model.from_pretrained` (transformers) — works, but is version-sensitive: we saw garbage output on transformers 5.10.1 (as of June 2026) and NaN logits when the original model was loaded first in the same process. Load in a fresh process and pin a DFloat11-supported transformers version.

Why DFloat11 looks slow in our numbers (and why that is GB10-local). The big slowdowns reported here — the custom online-decode engine at well under 1 tok/s, the transformers-path flakiness — are specific to this aarch64 GB10 box: there is no aarch64 vllm-df11 wheel, so we fell back to a per-forward Python decode path (eager, not a fused kernel), and the transformers decode kernel was unreliable on this library stack. On x86_64 with the vllm-df11 kernel, or a DFloat11-supported transformers build, DFloat11 runs closer to the ~2× envelope above, not these floor numbers.

Recorded VRAM (transformers path, after load): bf16 54.72 GB → DFloat11 37.30 GB — a 17.4 GB (−31.8%) weight saving, matching the 32% on-disk figure. (Three saving figures appear in this card, each measuring something slightly different: 17.4 GB weights-only here; ~14.5 GB in the four-config table below, which also holds an identical KV reservation across cells; and ~15.8 GB via the custom vLLM engine. The weights-only ~17 GB is the cleanest number.)

On-disk size:

[image]

Four-config comparison (one consistent harness)

All four configurations measured in a single engine on GB10 (so RAM and TPS are directly comparable):

[image]

Reading the plot: RAM — DFloat11 frees ~14.5 GB of resident memory (the bars include an identical KV-cache reservation across all four cells, so the gap is the weight saving). MTP adds a negligible amount. TPS — MTP is the speed lever (here 8.40/4.40 = 1.91× on bf16, vs 1.65× under vLLM above — same lever, different engine; it also lifted the DFloat11 path). This harness runs eager, per-forward decode for every cell, so its absolute rates sit below the vLLM CUDA-graph numbers above (e.g. bf16 here is 4.40 tok/s vs 5.7 tok/s under vLLM); the DFloat11 bars are especially low because each weight is decompressed per-forward in Python — that low rate is this box's integration overhead, not DFloat11 (DFloat11's real serving cost is the ~2.48× single-stream figure above, which amortizes toward parity with batch). All four configs produced byte-identical greedy output — so DFloat11's lossless compression and MTP's exact speculative decoding compose cleanly: you can stack both and still get bit-exact results.

A note on vLLM serving of the compressed weights

Where the vllm-df11 plugin is unavailable (e.g. aarch64), a custom online-decode path keeps the weights compressed in VRAM and decodes per-forward — bitwise-lossless vs bf16 up to batch-8 and freeing a measured −15.8 GB of resident weights. Treat it as a memory/correctness proof, not a throughput recommendation: 0.4–0.8 tok/s batched (~18–35× slower than bf16). That overhead is the integration (per-forward Python pre-hooks, eager mode, row-offset assembly), not DFloat11 itself. On bandwidth-bound hardware the real DFloat11 payoff is the smaller resident footprint (fit the model / more KV cache in less memory), not raw speed.

Speculative decoding (MTP) — how the head is included

The upstream Qwen/Qwen3.6-27B checkpoint ships *multi-token-prediction weights as top-level `mtp. tensors**. During compression we deliberately **kept that head out of the DFloat11 pass and carried it over verbatim as a raw bf16 sidecar** under mtphead/`, for two reasons: the head is small (so compressing it saves little), and the standard runtime model class did not expose matching MTP modules/state keys, so it could not be validated through the compressed path. A **lean download** (skip `mtphead/`, see Download above) simply omits these `mtp.` tensors for standard generation; the default download keeps them.

At serving time the head is consumed by vLLM's native speculative decoding on the bf16 base — speculative_config={"method":"mtp","num_speculative_tokens":1} (the exact config we tested, on vLLM 0.22 / GB10 — pin your vLLM version; see the vLLM recipe for Qwen/Qwen3.6-27B). It measured ~1.65× faster single-stream on GB10 (5.7 → 9.4 tok/s) — the most effective speed lever on this memory-bandwidth-bound box. We also confirmed MTP composes with the DFloat11 weights: in a single-engine test, DFloat11 + MTP produced byte-identical greedy output to DFloat11 alone, so lossless compression and exact speculative decoding stack cleanly. A lean download that omits the head cannot speculate, but is otherwise identical for standard decoding.

Provenance & correctness

• —Base model: `Qwen/Qwen3.6-27B` (BFloat16).
• —Compression: DFloat11 — lossless Huffman coding of the bf16 exponents; the sign and mantissa are untouched.
• —Strongest, hardware-independent guarantee: the DFloat11 artifact materializes back to bf16 bit-for-bit identical to the original checkpoint (verified by byte comparison). This is the rigorous sense in which the compression is lossless.
• —Runtime generation (clean process): greedy decoding matched the bf16 model token-for-token where we could run it cleanly — 8/8 reference prompts (full 48 tokens, up to batch-8) plus 8- and 32-token smoke tests. This was validated on the custom decode engine, not on the recommended DFloat11Model.from_pretrained (transformers) path (which showed version-sensitivity); 8/8 bit-exactness has not been independently re-verified on that path.
• —Runtime caveats on this GB10 / aarch64 box (kernel/library-local, not weight defects): a strict direct-forward suite that loaded the original model first and DFloat11 second in the same process produced NaN logits (clean DFloat11-only loading was finite); runtime logits are not claimed bit-identical for every prompt; and transformers==5.10.1 produced garbage. Load in a fresh process and pin a DFloat11-supported transformers version. On supported hardware/library versions these issues do not appear.

Citation

@inproceedings{zhang2025dfloat11,
  title     = {70\% Size, 100\% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float},
  author    = {Tianyi Zhang and Mohsen Hariri and Shaochen Zhong and Vipin Chaudhary and Yang Sui and Xia Hu and Anshumali Shrivastava},
  booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS)},
  year      = {2025},
  url       = {https://openreview.net/forum?id=xdNAVP7TGy}
}

Credits

• —DFloat11 — lossless LLM/DiT weight compression (Rice University & xMAD.ai).
• —`Qwen/Qwen3.6-27B` — base model.

Additional lossless compression experiments: rANS/range exponent coding

After publishing this DFloat11 checkpoint, we ran an offline follow-up experiment to test whether replacing DFloat11's Huffman-coded BF16 exponent stream with tighter entropy coders could reduce the model further while remaining exactly lossless. These experiments are checkpoint-format experiments only: they do not provide an inference-ready CUDA/vLLM runtime, and they do not change the mathematical model. They have also not been uploaded to HuggingFace.

What was tested

We kept the same lossless BF16 decomposition used by DFloat11:

• —sign bit + 7 mantissa bits stored exactly;
• —8-bit exponent stream entropy-coded;
• —reconstruction verified by byte-for-byte comparison against the upstream BF16 safetensors.

We tested two alternative coders over the raw exponent byte stream:

Losslessness

All completed alternate-coding runs reconstructed the original BF16 tensors exactly:

Size findings

The best raw-exponent result was range16, at about 286 MB smaller than this DFloat11 directory. rANS16 was essentially tied, about 286 MB smaller as well. Range coding beat rANS by only about 186 KB over the whole model.

The zstd19 payload pass was not useful: it made the process much slower and slightly larger. This suggests the exact sign/mantissa payload behaves close to incompressible byte noise for generic compression, so the remaining opportunity is not generic compression of mantissas.

Interpretation

The rANS/range experiments show that the current DFloat11/Huffman checkpoint is already close to the entropy limit for unconditional raw exponent coding. Better entropy coders alone give only a small additional reduction.

Predictive exponent-coding follow-up

We also ran a second offline follow-up to test whether the BF16 exponent stream could be made smaller before entropy coding by using cheap reversible predictors and coding the resulting residuals. This experiment kept the same lossless BF16 split as above: sign + mantissa bits were preserved exactly, and only the exponent stream was transformed and entropy-coded.

Stage 1: predictive residual artifact. We built a full predictive-rANS artifact using an automatic per-tensor choice among simple predictors:

• —raw exponent coding;
• —global tensor modal exponent;
• —previous-exponent residual;
• —left/nearby-column residual;
• —left-or-up residual;
• —per-row modal exponent;
• —per-tile modal exponent.

The resulting artifact was verified lossless across 1,199 / 1,199 tensors. It reached about 36.727 GB, making it the smallest tested lossless artifact so far, but only slightly smaller than raw rANS/range exponent coding:

In the auto-selected predictive artifact, most tensors still preferred raw exponent coding once predictor side information was counted:

Stage 2: predictor/context scout. We then ran an estimator over all 27,781,427,952 BF16 symbols to evaluate cheap context models before building more complex formats. The scout tested residual and conditional models using previous exponent, left/up neighboring exponent, per-row/per-tile modes, channel-index buckets, and related cheap metadata features.

The raw exponent entropy estimate was about 2.56926 bits/parameter. The best single scout model, a conditional tile-mode model, reached about 2.56850 bits/parameter, or only about 2.65 MB estimated savings over the whole model. Even an oracle-style per-tensor selection among the tested scout models gave only a small estimated gain.

Conclusion. These stages results suggest that, for this checkpoint, simple local or metadata-based exponent predictors do not expose a large additional compression opportunity. DFloat11-style storage appears close to the practical limit for the representation “exact sign/mantissa byte + entropy-coded BF16 exponent.”