Modelpublic

AaryanK/Muse-Glimmer-30B-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

21likes4.8kdownloads

Model Card

Muse-Glimmer-30B - GGUF (AK line)

👋 I built this line solo - the calibration, the per-tensor allocations, and the eval harness behind every number below. I'm looking for internships in AI agent orchestration and model inference. If this work looks relevant to your team: linkedin.com/in/theaaryankapoor

State-of-the-art GGUF quantizations for [`meta-models/Muse-Glimmer-30B`](https://huggingface.co/meta-models/Muse-Glimmer-30B). Eight builds (27.86 B params, 52 dense layers, GQA 32:2), each with a custom per-tensor bit allocation derived for its size point - plus the stock BF16 vision encoder.

Benchmarked head-to-head against the Unsloth, Meta and bartowski lines, every file scored on the same rig against the same BF16 reference: 26 wins, 6 statistical ties, 0 losses across 32 paired comparisons on two evaluation sets.

[image]

One line per publisher. Log y, bits-per-weight on the secondary axis, and the crowded 16 GB class magnified.

[image]

Every comparison with its 95 % interval - blue clears zero, grey is a statistical tie, and the right column carries the held-out C4 verdict. The full numbers are in the table below.

Comparison set: the three widest-distribution GGUF lines for this model, as published 2026-08-12; the Method section has everything needed to reproduce any number here.

File naming. Every quant in this line carries the AK- prefix: these are custom per-tensor allocations, not llama.cpp's stock recipes, so AK-Q4_K_M and a stock Q4_K_M are different files. mmproj keeps its upstream name.

Which file do I want?

file	size	bpw	mean KLD ↓	top-1 ↑	vs closest rival
`AK-Q2_K_XL`	12.45 GB	3.576	0.056036	90.88 %	−27 % KLD
`AK-Q3_K_XL`	13.51 GB	3.880	0.039079	92.30 %	−27 % KLD
`AK-Q4_K_M`	15.86 GB	4.556	0.013897	95.38 %	−6 % KLD
`AK-Q4_K_XL`	16.26 GB	4.669	0.012286	95.65 %	−14 % KLD
`AK-Q5_K_M`	19.19 GB	5.512	0.004974	97.26 %	best measured; −13 % vs Meta dynamic
`AK-Q6_K_XL`	26.24 GB	7.536	0.000876	98.82 %	−4 % KLD
`AK-Q8_K_L`	32.28 GB	9.272	0.000356	99.25 %	−21 % KLD, smaller file
`AK-Q8_K_XL`	34.96 GB	10.040	0.000316	99.30 %	most faithful build
`mmproj` BF16	3.85 GB	-	-	vision encoder	stock, unquantized

AK-Q4_K_XL is the strongest file in the crowded 16 GB class - no published quant of this model at any comparable size comes within 13 % of it, including Meta's own official quant, which it beats by 13 % while being half a GB smaller. The same story repeats at Q5: AK-Q5_K_M beats Meta's official 19.65 GB kquant-dynamic by 9-13 % on both evaluation sets while being 460 MB smaller. At Q8, AK-Q8_K_L beats Unsloth's build while being smaller.

bash

llama-server -m Muse-Glimmer-30B-AK-Q4_K_XL.gguf \
             --mmproj mmproj-Muse-Glimmer-30B-BF16.gguf -c 8192 -ngl 99

Full measurement table

publisher	file	bytes	bpw	PPL ratio	mean KLD	p99.9 KLD	top-1	Δ vs closest rival
bartowski	Q2KL	12,348,891,936	3.547	1.113088	0.132355	4.5183	86.289 %
Unsloth	UD-Q2KXL	12,444,212,256	3.574	1.065168	0.077057	2.9476	89.240 %
AaryanK	AK-Q2_K_XL	12,451,267,776	3.576	1.048590	0.056036	2.4523	90.878 %	−27.3 % [−28.3, −26.3]
Unsloth	UD-Q3KXL	13,360,983,072	3.837	1.047022	0.053173	2.2479	91.175 %
AaryanK	AK-Q3_K_XL	13,509,095,872	3.880	1.033090	0.039079	1.5369	92.303 %	−26.5 % [−27.8, −25.2]
bartowski	Q3KM	13,962,519,328	4.010	1.032023	0.039487	1.6390	92.303 %
bartowski	IQ4_XS	15,435,096,096	4.433	1.010121	0.015440	0.6246	95.128 %
AaryanK	AK-Q4_K_M	15,864,857,280	4.556	1.010896	0.013897	0.5592	95.378 %	−5.6 % [−7.6, −3.3]
Unsloth	UD-Q4KXL	15,878,222,368	4.560	1.010630	0.014714	0.5871	95.249 %
AaryanK	AK-Q4_K_XL	16,255,873,984	4.669	1.008957	0.012286	0.5601	95.647 %	−14.0 % [−15.4, −12.7]
bartowski	Q4KS	16,320,943,136	4.687	1.010071	0.014293	0.5860	95.319 %
Meta	kquant-17gb	16,756,681,056	4.812	1.009871	0.014146	0.5918	95.297 %
AaryanK	AK-Q5_K_M	19,191,472,832	5.512	1.003958	0.004974	0.1922	97.256 %	−2.3 % [−4.8, +0.6] tie
Unsloth	UD-Q5KM	19,194,274,848	5.513	1.004517	0.005092	0.2027	97.157 %
Meta	kquant-dynamic	19,653,957,984	5.645	1.004101	0.005687	0.2206	96.965 %
AaryanK	AK-Q6_K_XL	26,238,366,400	7.536	1.000831	0.000876	0.0384	98.819 %	−3.8 % [−5.8, −1.9]
Unsloth	UD-Q6KXL	26,265,362,976	7.543	1.000885	0.000911	0.0386	98.867 %
AaryanK	AK-Q8_K_L	32,283,878,048	9.272	1.000606	0.000356	0.0148	99.248 %	−20.8 % [−23.8, −17.5]
Unsloth	UD-Q8KXL	32,300,651,040	9.277	1.000728	0.000450	0.0197	99.126 %
AaryanK	AK-Q8_K_XL	34,958,791,360	10.040	1.000680	0.000316	0.0140	99.301 %	−29.7 % [−32.4, −26.6]

Intervals are a paired per-token cluster bootstrap over the 60 evaluation chunks. Δ is against the closest-sized non-AaryanK file.

Reading the numbers

Two Q8 builds, two jobs. AK-Q8_K_L is the size-class winner - smaller than Unsloth's Q8 and −20.8 % KLD, confirmed on every slice tested. AK-Q8_K_XL is the maximum-fidelity build: −29.7 % at +8.2 % bytes (10.04 bpw vs 9.28), for when the last 2.7 GB of VRAM is cheaper than the last drop of divergence.

PPL ratio and KLD can rank differently at Q3-Q4. PPL scores only the probability of the true next token; KLD scores the whole distribution - so AK-Q3_K_XL and AK-Q4_K_M lead their size peers on KLD while trailing by ~0.001 on PPL ratio. Both metrics are in the table; the interval column carries the verdicts.

Does the margin generalise?

[image]

The same files re-measured on six evaluation sets, four held out and audited at zero fragment overlap with any calibration corpus. 11 of 16 held-out margins exceed the same file's wikitext margin, and every interval in the chart excludes zero - including all four held-out domains for AK-Q8_K_L, where the size-matched Q8 lead spans −17 % to −24 %.

Tail behaviour

[image]

Method

—Reference: our own BF16 GGUF, converted with llama.cpp pinned at 62bf73d2. The conversion was checked against every competitor's file across 21 load-bearing KVs, so the comparison measures quantization rather than a conversion delta.
—Eval: llama-perplexity --kl-divergence, ctx 4096 × 60 chunks → 122,820 scored tokens. ctx 4096 matters for this architecture - it alternates 3× sliding-window (2048) with 1× full-attention NoPE layers, and only at ctx ≥ 4096 does every scored token sit beyond the window.
—Statistics: paired per-token cluster bootstrap at the 2047-token chunk width for every interval.
—Confirmation: the Q4 result was re-run under 9 independent calibration draws across three corpus families on an untouched slice - beneficial in 9/9, no reversals, every interval excluding zero, against an MDE fixed before any data was collected.
—Long context: re-measured at ctx 8192; the lead holds and slightly grows.
—Capability: a 130-case tool-calling suite scored as paired agreement with BF16 - AK-Q4_K_M matches BF16 on 128 of 130 cases with one flip in each direction: statistically indistinguishable (exact McNemar p = 1.000).
—Ten pre-registered apparatus gates, all passing, including full-vocabulary agreement with HF transformers and exact greedy generation agreement (235/235 tokens).
—Scope: the text tower is what is measured and quantized; mmproj ships as the stock BF16 encoder. KLD values are model-local (this head applies logit soft-capping) - compare within this table only.

Held-out sets: GitHub source (numpy/redis/django/sqlite/nlohmann), OASST dialogue, GSM8K + arXiv abstracts, and Wikipedia in sixteen non-English languages.

Per-tensor bit allocation derived separately at each target bit-width using importance data from a diverse in-house calibration set. Base model licence and usage policy unchanged from [`meta-models/Muse-Glimmer-30B`](https://huggingface.co/meta-models/Muse-Glimmer-30B) (Apache-2.0).