ProCreations/MiMo-V2.6-Flash-RL-IQ3_XXS-GGUF

MiMo-V2.6-Flash-RL: calibrated IQ3_XXS

This is a lossy, importance-calibrated IQ3_XXS conversion of the original MXFP4 routed expert weights. It prioritizes retention: all non-expert tensors keep their original decoded values, using FP32 or exactly representable BF16 storage. Calibration and evaluation use the original released checkpoint as the reference, not another 3-bit model or a dequantized NVFP4 approximation.

Source: XiaomiMiMo/MiMo-V2.6-Flash-RL, revision 3b38d063180c3e4aed9691fdc735f3d10b266ee4, MIT license. For exact preservation of all released weight values, see the separately verified NVFP4 conversion.

Calibration

The importance matrix is collected from the original MXFP4 experts with cuBLAS F32 storage/accumulation and BF16 KV, bypassing the fast runtime's FP4/Q8 activation quantizers. The cuBLAS handle permits TF32 arithmetic for eligible large multiplications, so this is not a full-mantissa FP32 calibration claim. The selected corpus contains 1,051,701 base tokens plus 167,691 extension tokens. After serialization and runtime tokenization, collection processes 257 chunks at 4,096 context and 10 chunks at 16,384 context: 1,216,512 tokens. All 141 expert projection count sums independently confirm eight routes per processed token. The extension adds complete long code and math examples and deterministic tool/JSON transcripts with checked answers.

Public source datasets are OpenCodeReasoning, OpenR1-Math-220k, ultrachat_200k, and alpaca-gpt4-chinese. Their immutable revisions, selection seeds, complete-example filters and hashes are recorded in the manifests. Question-hash splits and global deduplication separate calibration from the held-out code, math, English and Chinese evaluation text. Executable coding checks are separate handcrafted tasks, not calibration examples.

This is an explicitly mixed-type release: 129 of 141 routed-expert projection matrices use IQ3_XXS. The 12 matrices in layers 1, 7, 11 and 25 retain their original MXFP4 representation because at least one expert did not meet the 256-observation coverage floor. All 649 model tensors passed the shape/type and preservation audit. The importance matrix is included; its SHA256 is 7a82cf559878c59e7a61075608db3cd7fc0c7677354d9d932be6daec6eaadfb7.

The original MXFP4 weights were dequantized by the quantizer and requantized to IQ3_XXS using that matrix. This is lossy. The exact NVFP4 transcode was not used as an intermediate. No uncalibrated routed expert was silently forced to IQ3. The full recipe and per-expert coverage are in reports/.

Measured retention

The matched four-domain text comparison evaluates 65,504 next-token positions from held-out code, math, English and Chinese text (eight 4,096-token chunks per domain, scoring each chunk's second half). Both models use the same cuBLAS F32 storage/accumulation, TF32-permitted arithmetic and BF16 KV settings.

Lower PPL on English/Chinese does not establish higher task accuracy or identical behavior; the output distributions changed. In particular, 83.531% Chinese top-token agreement is not an extremely-high-retention guarantee. This release is calibrated for retention, but broad downstream task evaluation was not completed before the cloud conversion job was closed. See `heldout-iq3-strict.json` for the numerical results and the precise scope of its completion status.

These measurements describe this candidate and protocol. Perplexity ratios and top-token agreement are not percentages of general intelligence or task accuracy. The saved llama.cpp baseline logits are clipped and uint16 encoded; perplexity ratios use the original unclipped PPL logs. Runtime kernels, KV dtype, activation quantization and sampling are reported separately from weight quantization.

Loading and sampling

The eight target shards total 141,408,349,312 bytes (131.697 GiB). Keep all eight in the same directory and load the first shard. Download with:

hf download ProCreations/MiMo-V2.6-Flash-RL-IQ3_XXS-GGUF --local-dir mimo-iq3
cd mimo-iq3
bash tools/build_workstation_runtime.sh

The patched runtime targets llama.cpp revision 58367713a6935c0810103378144008df32e3d5db. A reference-style CUDA build bypasses low-bit activation kernels:

cmake -S llama-mimo-nvfp4 -B llama-mimo-nvfp4/build-reference -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DGGML_CUDA_FORCE_CUBLAS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120 -DLLAMA_CURL=OFF
cmake --build llama-mimo-nvfp4/build-reference -j 16 --target llama-server
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 GGML_CUDA_DISABLE_GRAPHS=1 \
  ./llama-mimo-nvfp4/build-reference/bin/llama-server \
  -m gguf/MiMo-V2.6-Flash-RL-IQ3_XXS-00001-of-00008.gguf \
  -ngl 999 --tensor-split 0.47,0.53 -c 4096 -b 512 -ub 512 \
  --flash-attn on --cache-type-k bf16 --cache-type-v bf16 \
  --temp 1 --top-p 0.95 --top-k 0 --min-p 0 \
  --fit off --parallel 1 --host 127.0.0.1 --port 18085

This reference-style recipe assumes enough total GPU memory, such as the two 96GB GPUs used for validation. CPU offload is possible, but single-card speed, long context and speculative decoding are not qualified by this IQ3 release. Changing kernels, KV precision, or TF32 settings changes the numerical profile.

Use temperature 1.0 and topp 0.95, with topk=0 and min_p=0 to avoid additional runtime-default truncation. Allow enough output tokens for reasoning and count truncated responses as incomplete. The original DFlash weights are reusable from the NVFP4 repository; their inclusion does not guarantee support in every runtime. This text-oriented GGUF release does not certify multimodal generation.

Reproduction

The tools/ directory includes the pinned runtime patches, original-weight GGUF conversion, calibration selection/collection, recipe generation, quantizer invocation, and independent artifact verification. reports/ records upstream and dataset revisions, corpus hashes, actual calibration consumption, the quantization recipe, held-out measurements, and every published shard SHA256.

After quantization, 103 original FP32 matrices were stored as BF16 only after checking that every value was exactly representable. Independent readback restored their original FP32 bit patterns. All other preserved tensors retain the original stored bytes. See `gguf-release-verification.json`. Runtime patches and experimental tuning options are supplied for reproduction; including an option is not a claim that it improves IQ3 accuracy or performance.