kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF

### ⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL bailingmoe3 is not merged upstream, and Ling-3.0-tiny additionally needs the Q-LoRA attention path (q_lora_rank: 256) that flash-era builds lack. Ignore the auto-generated "Use this model" commands above — use the patch in patches/. 🚀 101.08 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) — 4.30 GiB, smaller than Q4KM (4.49 GiB). Verified on two independent machines.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to `charlie12345/ROCmFPX` at commit `2809dc5`, verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: `patches/README.md` in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB is configure-time, so cmake --build alone will not link them.

Ling-3.0-tiny — ROCmFP4 (tier 102 COHERENT) GGUF

A 4-bit ROCmFP4 quantization of `inclusionAI/Ling-3.0-tiny`, built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo) with per-tensor protection of the LM head, token embeddings, shared experts, router, and the hybrid model's recurrent state.

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

Two independent reasons, both unavoidable:

1. 1.`bailingmoe3` is not in upstream llama.cpp. Support is still open in PR #26608 (unmerged at time of writing).
2. 2.Ling-3.0-tiny needs the Q-LoRA attention path. Its config sets q_lora_rank: 256 (q_a_proj → q_a_layernorm → q_b_proj). Several existing bailingmoe3 implementations were written against Ling-3.0-flash, which has q_lora_rank: null and therefore no query compression. On such a build, every GGUF of tiny fails — including the BF16 and Q4KM ones — typically at missing tensor 'blk.0.ssm_f.weight', before the Q-LoRA gap is even reached.

You need a build with both bailingmoe3 and its Q-LoRA path. The reference implementation is the branch behind PR #26608 (aetherbird/llama.cpp, branch bailingmoe3-support). The ROCmFP4 quant types additionally require a fork that implements them; upstream llama.cpp does not have Q4_0_ROCMFP4_*.

If your build loads Ling-3.0-flash but not tiny, you are missing the Q-LoRA path specifically.

⚠️ strings is not a capability check

We tested a second gfx1151 machine whose libllama.so contained `bailingmoe3` (60 matches), `ssm_f_a`, and `attn_q_a` — it looked fully capable. It still failed with the exact same missing tensor 'blk.0.ssm_f.weight'.

Those symbols live in the tensor-name table. The fallback logic that maps ssm_f → ssm_f_a, and the Q-LoRA branch itself, are separate code. Grepping the binary tells you nothing — attempt the load.

All quant variants

Three builds of this model, all measured in one session on one box with one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4, ROCmFPX-2809dc5) — so these rows are directly comparable. Median of 3, warm-up discarded, otherwise-idle box.

⚠️ The 4-bit build is faster (104.04 vs ~89 tok/s) and 44% smaller. These 8-bit builds exist for accuracy headroom, not speed — pick them only if you need the extra precision.

What `AGENT` actually changes: it keeps far more tensors at true Q8_0 instead of the packed 8-bit type — measured in these files, 135 tensors vs 2 tensors. On models with an MTP draft head that raises draft acceptance and wins ~6%; these two models have no MTP head, and here the two 8-bit builds are within noise of each other.

Measured results

Verified on two independent gfx1151 machines, using the model's official sampling (temperature 0.6, top_p 0.95, top_k 20).

⭐ The build is portable across ROCm minor versions

The binaries were compiled against ROCm 7.2.4 and run unmodified on a ROCm 7.13.0 host — all 9 Q4_0_ROCMFP4_* quant types still enumerated, model loads, 101 tok/s. Both hosts are gfx1151. Useful if you have one build box and several Strix Halo machines: you can copy llama-server + lib*.so* rather than rebuilding per host (set LD_LIBRARY_PATH to the directory you copied them into).

Serving configuration that works

Long-running deployment on machine B (systemd, always-hot):

-dev ROCm0 -ngl 999 -fa on -c 32768 -fit off -np 1 --no-mmap --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

plus LimitMEMLOCK=infinity, HSA_OVERRIDE_GFX_VERSION=11.5.1, GGML_HIP_ENABLE_UNIFIED_MEMORY=1.

Verified on the running process, not just at launch: memlock unlimited (the 8 MB default will hobble the model), `n_ctx = 32768` actually granted — --fit is on by default and can silently shrink context or push tensors to CPU, so -fit off and then confirm the number — and zero "tensor override to CPU" lines in the log.

Per-tensor protection (audited in the finished file)

Why this matters. Tier _STRIX (105) protects attention K/V but not the LM head — on this model's 157,184-token vocabulary that leaves every logit passing through a 4-bit tensor. Tier 102 COHERENT carries Q6_K token embeddings, and the head/shared-expert protections above were applied explicitly. Shared experts matter because they are dense — they process every token, so their error is systematic rather than averaged across the 128 routed experts.

The recurrent/linear-attention state (ssm_a, ssm_dt, conv1d) is kept at F32: these are float32 in the source model, and quantizing hybrid state is a known way to produce a model that loads, runs, and emits fluent nonsense.

Size comparison (same source, same machine)

What was NOT measured

Stated plainly so you can judge fitness for your use case:

• —No perplexity run, and no quality A/B against Q4_K_M or BF16. The correctness checks above are memorized-fact prompts — they are necessary but not sufficient, and a damaged model can pass them.
• —No long-context testing. All generations were short. The 32,768-token context was granted and confirmed at load on machine B, but nothing exercised rope/KV behaviour at depth, and nothing was run near the model's 131,072 ceiling.
• —No tool-calling evaluation.
• —MTP / speculative decoding untested — Ling-3.0-tiny has num_nextn_predict_layers: 0, so it has no MTP layer to exercise.

Model

BailingMoeV3ForCausalLM / bailing_hybrid, GGUF arch bailingmoe3. 24 layers in a 3:1 stack of KDA (Kimi Delta Attention, 18 layers) and MLA (Multi-head Latent Attention, 6 layers) · hidden 1536 · 128 routed experts, 8 active · shared experts · vocab 157,184 · q_lora_rank 256 · kv_lora_rank 512 · context 131,072.

Base model licence: MIT (inherited). All credit for the model itself goes to inclusionAI.