Modelpublic

wepiqx/Ling-3.0-tiny-ASHQ1-GGUF

sourceHugging Faceapache-2.0updated 17d agoView on Hugging Face

0likes811downloads

Model Card

Ling-3.0-tiny — ASHQ1 Quantization (Top-Down)

ASHQ1 quantization of Ling-3.0-tiny (7.9B total / 1.3B active, 128 routed + 1 shared experts, KDA+MLA hybrid attention) with the top-down allocator: every tensor starts at F16 and is greedily downgraded cheapest-loss-first until the budget fits.

MoE-specific handling: routers (ffn_gate_inp, exp_probs_b) pinned at F16 outside the budget, shared experts mapped to high tiers (always active), routed experts graded by depth (early layers Q4K → late layers Q6K/Q8_0).

Note: File names contain "BF16" for HuggingFace parser compatibility — these are ASHQ1 quants, not BF16.

Quants

File	Size	PPL (wiki.test.raw, ctx 1024)	Description
`Ling-3.0-tiny-BF16-ASHQ1-5000.gguf`	4861 MiB	13.0681 ± 0.1051	Bottom-up, best of the duel — smaller and better
`Ling-3.0-tiny-BF16-TOPDOWN-5000.gguf`	4999 MiB	13.1094 ± 0.1054	Top-down challenger (local only)
`Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf`	6013 MiB	13.0001 ± 0.1047	Top-down, first MoE quant
`Ling-3.0-tiny-BF16-TOPDOWN-6000-v7.1.gguf`	6016 MiB	12.9802 ± 0.1045	Top-down + phase-2 slack filling (local only)
`Ling-3.0-tiny-i1-Q6_K.gguf`	6198 MiB	13.0123 ± 0.1049	Uniform Q6_K with imatrix (reference, local only)

Bottom-up vs top-down (@5000, same budget)

Two opposite allocations, same PPL within noise (Δ=0.04, σ=0.10):

Tier	Bottom-up (4861 MiB)	Top-down (4999 MiB)
F16	215 / 9.7 MiB	126 / 222.8 MiB
Q4_K	43 / 2271 MiB	210 / 2059 MiB
Q5_K	30 / 2036 MiB	36 / 2304 MiB
Q6_K	8 / 94 MiB	14 / 109 MiB
Q8_0	230 / 432 MiB	140 / 285 MiB

Bottom-up spreads upgrades wide (241 small tensors at Q8_0); top-down keeps the most precious tensors at F16 (incl. top-importance shared experts) and crushes the rest to the floor. Lesson: PPL can't separate the two paths — the bit distribution differs radically, the quality doesn't. Finer-grained tasks (HumanEval, MMLU) may.

Speed

MoE sparsity pays off: only 1.3B of 7.9B params active per token. On a GTX 1070 (8 GB) the quant runs at ~76 t/s mixed load and fits the full 131072-token context without problems.

Distribution (@6000, top-down)

Tier	Tensors	Notes
F16	248	pinned norms/routers + untouched high-importance
Q4_K	183	early routed experts, floors
Q5_K	23	incl. output/token_embd pins
Q6_K	41	late routed experts
Q8_0	31	shared experts, MLA factors

Usage

llama.cpp

bash

llama-cli \
  -m Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf \
  --jinja \
  -ngl 99 \
  -c 8192

Recommended sampling: temperature 1.0, top_p 0.95, top_k 20.

Ollama

FROM ./Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf

PARAMETER num_ctx 8192
PARAMETER temperature 1.0
PARAMETER top_k 20
PARAMETER top_p 0.95

bash

ollama create ling-tiny-topdown-6000 -f Modelfile
ollama run ling-tiny-topdown-6000

Quantization Config (@6000, top-down)

Generated by ASHQ1 top-down. Reproduce with: python main.py --model Ling-3.0-tiny-bf16.gguf --imatrix Ling-3.0-tiny-imatrix.gguf --size 6000 --top-down --run

<details> <summary>llama-quantize arguments (click to expand)</summary>

bash

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(23)\.ffn_down_exps=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_gate=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_up=Q8_0"
--tensor-type "(blk|BLK)\.((?:2|3|4|5|6|7|8|9|10|11|12|13))\.ffn_down_shexp=Q8_0"
--tensor-type "(blk|BLK)\.(1)\.ffn_down_shexp=Q6_K"
--tensor-type "(blk|BLK)\.((?:6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21))\.attn_output=Q8_0"
--tensor-type "(blk|BLK)\.((?:17|18|19|20|21|22))\.ffn_down_exps=Q6_K"
--tensor-type "(blk|BLK)\.(0)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_exps=Q6_K"
--tensor-type "(blk|BLK)\.([2-5])\.attn_output=Q6_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_exps=Q6_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_up_exps=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_gate_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:14|15|16|17|18|19|20|21|22|23))\.ffn_down_shexp=F16"
--tensor-type "(blk|BLK)\.((?:12|13|14|15|16))\.ffn_down_exps=Q5_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11))\.ffn_down_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.exp_probs_b=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_inp=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_beta=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_k=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_mqa=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_b=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_q=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_f_a=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_v=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_g_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_gate=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_k_b=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_v_b=F16"
--tensor-type "(blk|BLK)\.((?:22|23))\.attn_output=F16"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_up_exps=Q5_K"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_gate_exps=Q5_K"
--tensor-type "(blk|BLK)\.([0-1])\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_k=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_a=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_dt=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.attn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_v=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_q=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a_norm=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_norm=Q4_K"
--tensor-type ".*output_norm.*=Q4_K"

</details>

Notes

—Source BF16: inclusionAI/Ling-3.0-tiny (24 layers, bailingmoe3 arch, no MTP)
—Imatrix: custom Ling-3.0-tiny imatrix
—MoE: 128 routed experts (8 active) + 1 shared expert per token, expert FF 512 (no K-quant padding needed)
—Quantization and tuning by wepiqx
—Built with llama.cpp
—License: Apache-2.0 (upstream)