0xKitkat/Qwen3.8-Flash-Next-125B-UltraLite-37GiB-GGUF

⚡ Qwen3.8 Flash Next 125B — UltraLite 37 GiB GGUF

125B parameters. 37 GiB. One wild GGUF.

This experimental text-only build compresses Qwen3.8-Flash-Next from a 72.55 GB calibrated source GGUF to 39,721,239,200 bytes:

• —36.993287 GiB
• —39.721239 decimal GB
• —1.80 BPW tensor payload
• —125B total parameters / approximately 6B active parameters
• —278,760,800 bytes below a strict 40,000,000,000-byte ceiling

The release is not a blind whole-model requant. It uses a patched GGUF layout, streaming row repacking, mixed quantization by tensor role, boundary-layer protection, byte-for-byte preservation of sensitive paths, and structural removal of optional components not used by ordinary text generation.

Follow the experiments

If you want more aggressively compressed models, weird inference experiments, and build notes, follow [@procrastiness on X](https://x.com/procrastiness).

Important runtime requirement

This model uses a padded physical representation for per_layer_token_embd.weight. It therefore requires the included patches/qwen4exp-under40.patch applied to the pinned Qwen4Exp llama.cpp revision. Stock llama.cpp does not understand this custom physical/logical row layout yet.

The patch is part of the release, and the exact build/runtime revision is pinned in build-manifest.txt.

What was compressed

The source is Unsloth's UD-IQ1_S release at revision 83cadfda58d30be06c110518208d1bb918b33f10. Its three source shards total 72,546,461,344 bytes, with 72,535,436,800 bytes of tensor data.

Boundary-layer protection

Expert gate and up projections use Q2_0 in layers:

0, 1, 2, 44, 45, 46, 47

Layers 3–43 use Q10 for those expert banks. All 48 expert down banks remain Q20. This spends the final byte budget on the transformer entrance and exit, where error propagation is especially undesirable.

The 160 → 256 padded PLE architecture

The per-layer embedding table contains 320,001,536 logical rows of 160 values. The target ultra-low-bit block layout cannot directly encode that row width. The included llama.cpp patch therefore:

1. 1.Reads the source table in bounded chunks of 32,768 rows.
2. 2.Decodes each 160-value logical row.
3. 3.Appends 96 zeros, producing a 256-value physical row.
4. 4.Quantizes that physical row to Q1_0.
5. 5.Stores the physical width in GGUF tensor metadata.
6. 6.At inference, gathers the padded physical row.
7. 7.Creates a strided view of its first 160 values.
8. 8.Materializes the original logical shape before the normal Qwen4Exp graph consumes it.

The padding does not prune learned PLE values: every original element remains present. Only the 96 appended positions are zeros. Quantization error still exists, as expected from Q1_0.

Structural pruning

• —The optional vision projector is not included; this is a text-only release.
• —The normal-generation source GGUF does not contain the separate 4B MTP training/speculation companion head.
• —No ordinary text-generation transformer layer was removed.
• —No attention, router, shared-expert, normalization, SSM, indexer, token embedding, or output tensor was pruned.

Exact result

All 1,224 tensors converted successfully. The patched runtime loaded the three shards, accepted the padded PLE representation, evaluated a prompt, generated a deterministic first token, and exited with code 0. Smoke-test stderr was empty.

Build method

The reproducible pipeline is included as scripts/remote_build.sh. It performs the following complete sequence:

1. 1.Pins the source model repository and exact source revision.
2. 2.Downloads all three source shards with resumable parallel transfers.
3. 3.Validates every source shard against its exact expected byte count.
4. 4.Checks out Unsloth's Qwen4Exp llama.cpp branch at commit 250b61446efc91e3a179c8677956f2667c8fbda0.
5. 5.Applies patches/qwen4exp-under40.patch.
6. 6.Builds llama-quantize, llama-cli, and llama-perplexity.
7. 7.Executes an exact dry-run allocation with explicit per-tensor overrides.
8. 8.Refuses conversion unless projected payload plus 32 MiB of GGUF overhead is below 40,000,000,000 bytes.
9. 9.Streams and requantizes only the selected tensors; unchanged tensors are copied byte-for-byte.
10. 10.Keeps the source three-shard split.
11. 11.Enforces the hard size limit against final filesystem byte counts.
12. 12.Runs a deterministic non-interactive generation smoke test.
13. 13.Generates SHA-256 checksums and a complete build manifest.

The effective quantization command is:

llama-quantize \
  --allow-requantize \
  --keep-split \
  --tensor-type '^per_layer_token_embd[.]weight$=Q1_0' \
  --tensor-type '^blk[.](0|1|2|44|45|46|47)[.]ffn_(gate|up)_exps[.]weight$=Q2_0' \
  --tensor-type '^blk[.][0-9]+[.]ffn_down_exps[.]weight$=Q2_0' \
  --tensor-type '^blk[.][0-9]+[.]ffn_(gate|up)_exps[.]weight$=Q1_0' \
  SOURCE-00001-of-00003.gguf \
  Qwen3.8-Flash-Next-125B-UltraLite-37GiB \
  COPY \
  "$(nproc)"

COPY is intentionally used as the default. The patch permits explicit tensor overrides on top of COPY, so everything not matched by the four expressions retains its original calibrated encoding.

Build it yourself

Required tools on Linux:

sudo apt-get update
sudo apt-get install -y git git-lfs cmake build-essential curl aria2

Then place the patch beside the script and run:

mkdir -p /root/qwen38-compress
cp scripts/remote_build.sh /root/qwen38-compress/
cp patches/qwen4exp-under40.patch /root/qwen38-compress/
chmod +x /root/qwen38-compress/remote_build.sh
/root/qwen38-compress/remote_build.sh

The process needs roughly 73 GB for the source, 40 GB for the output, build space, and temporary headroom. The custom PLE conversion is chunked so it does not allocate the full decoded table in memory.

Run the model

Clone and patch the pinned runtime:

git clone --branch qwen4exp/qwen3.8-flash-next \
  https://github.com/unslothai/llama.cpp.git
cd llama.cpp
git checkout 250b61446efc91e3a179c8677956f2667c8fbda0
git apply ../patches/qwen4exp-under40.patch

cmake -S . -B build -DGGML_CUDA=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j "$(nproc)" --target llama-cli

Run from the directory containing all three GGUF shards:

./llama.cpp/build/bin/llama-cli \
  -m Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf \
  -c 4096 \
  -t "$(nproc)"

The build was validated CPU-only. GPU backends can be added to the patched runtime in the normal llama.cpp build configuration when supported by the host.

Perplexity comparison

Use the same corpus, context, batch sizes, and runtime for both source and UltraLite:

LLAMA=./llama.cpp/build/bin/llama-perplexity
SRC=/path/to/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf
DST=/path/to/Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf
CORPUS=/path/to/calibration.txt

"$LLAMA" -m "$SRC" -f "$CORPUS" -c 2048 -b 128 -ub 128 -t "$(nproc)"
"$LLAMA" -m "$DST" -f "$CORPUS" -c 2048 -b 128 -ub 128 -t "$(nproc)"

Reproducibility files

• —build-manifest.txt — exact sizes, revisions, format choices, smoke result, and checksums
• —SHA256SUMS — shard integrity hashes
• —patches/qwen4exp-under40.patch — converter and runtime support
• —scripts/remote_build.sh — complete resumable build pipeline
• —scripts/deploy_to_triologay.ps1 — Tailscale deployment helper used for the original build

Credits

• —Qwen for the model family
• —Unsloth's Qwen3.8-Flash-Next GGUF release
• —llama.cpp
• —Qwen4Exp llama.cpp work

Built as an aggressive compression experiment. For the next one, follow [@procrastiness on X](https://x.com/procrastiness).