aleada/pack-integrity-check
0
1<!doctype html>
2<html lang="en">
3<head>
4<meta charset="utf-8">
5<meta name="viewport" content="width=device-width, initial-scale=1">
6<title>Does this pack keep what it claims?</title>
7<!-- The Space renders inside an iframe and huggingface.co sends
8 x-frame-options: DENY, so any link without a target tries to load
9 a refused page in the frame and reads as broken. Defaulting every
10 link out of the frame fixes the one that was, and the next one
11 nobody remembers to mark. -->
12<base target="_blank">
13<style>
14 :root {
15 --bg: #fbfbfa;
16 --panel: #ffffff;
17 --ink: #1a1a19;
18 --muted: #6b6b66;
19 --line: #e4e3df;
20 --accent: #3d5a80;
21 --problem: #a03030;
22 --problem-bg: #fdf3f2;
23 --check: #8a6212;
24 --check-bg: #fdf9ee;
25 --clear: #2f6b45;
26 --clear-bg: #f1f8f3;
27 --mono: ui-monospace, "SF Mono", "Cascadia Mono", Menlo, monospace;
28 }
29 @media (prefers-color-scheme: dark) {
30 :root {
31 --bg: #16171a;
32 --panel: #1d1f23;
33 --ink: #e8e6e3;
34 --muted: #9a978f;
35 --line: #2e3137;
36 --accent: #8ab0d9;
37 --problem: #e08a86;
38 --problem-bg: #2a1e1e;
39 --check: #d9b76a;
40 --check-bg: #29241a;
41 --clear: #8fc9a6;
42 --clear-bg: #1b2620;
43 }
44 }
45
46 * { box-sizing: border-box; }
47 body {
48 margin: 0;
49 background: var(--bg);
50 color: var(--ink);
51 font: 16px/1.65 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif;
52 -webkit-font-smoothing: antialiased;
53 }
54 .wrap { max-width: 50rem; margin: 0 auto; padding: 3.5rem 1.25rem 5rem; }
55
56 /* The mark is white-on-transparent — made for a dark background, so
57 invisible on the light one. It is 100% monochrome, checked pixel
58 by pixel, so a straight inversion is clean rather than a
59 colour-shifted approximation. */
60 .brand { display: inline-block; margin-bottom: 1.75rem; }
61 .brand img {
62 height: 1.5rem; width: auto; display: block;
63 filter: invert(1); opacity: .78;
64 }
65 @media (prefers-color-scheme: dark) {
66 .brand img { filter: none; opacity: .9; }
67 }
68
69 header h1 {
70 font-size: clamp(1.6rem, 4.5vw, 2.2rem);
71 line-height: 1.2;
72 letter-spacing: -0.02em;
73 margin: 0 0 .75rem;
74 font-weight: 620;
75 }
76 header p { color: var(--muted); margin: 0 0 .5rem; max-width: 42rem; }
77
78 form {
79 display: flex; gap: .5rem; flex-wrap: wrap;
80 margin: 2rem 0 .75rem;
81 }
82 input[type=text] {
83 flex: 1 1 20rem; min-width: 0;
84 padding: .7rem .85rem;
85 border: 1px solid var(--line); border-radius: .5rem;
86 background: var(--panel); color: var(--ink);
87 font: inherit; font-family: var(--mono); font-size: .92rem;
88 }
89 input[type=text]:focus {
90 outline: 2px solid var(--accent); outline-offset: -1px; border-color: transparent;
91 }
92 button {
93 padding: .7rem 1.4rem; border: 0; border-radius: .5rem;
94 background: var(--accent); color: #fff;
95 font: inherit; font-weight: 560; cursor: pointer;
96 }
97 button:hover { filter: brightness(1.08); }
98 button:disabled { opacity: .55; cursor: progress; }
99
100 .examples {
101 font-size: .86rem; color: var(--muted); margin-bottom: 2.5rem;
102 line-height: 2;
103 }
104 .examples .group { display: inline-block; min-width: 5.5rem; }
105 .examples button {
106 background: none; color: var(--accent); padding: 0 .15rem;
107 font-size: .86rem; font-family: var(--mono); text-decoration: underline;
108 text-underline-offset: 2px; font-weight: 400;
109 }
110
111 .card {
112 background: var(--panel); border: 1px solid var(--line);
113 border-radius: .7rem; padding: 1.1rem 1.3rem; margin: .85rem 0;
114 }
115 .card h3 {
116 margin: 0 0 .5rem; font-size: 1.02rem; font-weight: 600;
117 display: flex; gap: .55rem; align-items: baseline;
118 }
119 .card p { margin: .5rem 0; }
120 .card ul { margin: .5rem 0; padding-left: 1.1rem; }
121 .card li { font-family: var(--mono); font-size: .84rem; margin: .15rem 0; }
122
123 .problem { border-left: 3px solid var(--problem); background: var(--problem-bg); }
124 .problem h3 { color: var(--problem); }
125 .check { border-left: 3px solid var(--check); background: var(--check-bg); }
126 .check h3 { color: var(--check); }
127 .clear { border-left: 3px solid var(--clear); background: var(--clear-bg); }
128 .clear h3 { color: var(--clear); }
129
130 .meta {
131 display: flex; flex-wrap: wrap; gap: .35rem 1.5rem;
132 font-size: .86rem; color: var(--muted);
133 padding-bottom: .9rem; margin-bottom: .3rem;
134 border-bottom: 1px solid var(--line);
135 }
136 .meta code { color: var(--ink); }
137 code { font-family: var(--mono); font-size: .86em; }
138
139 h2 {
140 font-size: 1.05rem; font-weight: 600; letter-spacing: -0.01em;
141 margin: 3rem 0 .75rem;
142 }
143 table { border-collapse: collapse; width: 100%; font-size: .87rem; }
144 th, td {
145 text-align: left; padding: .45rem .7rem .45rem 0;
146 border-bottom: 1px solid var(--line); vertical-align: top;
147 }
148 th { font-weight: 600; color: var(--muted); font-size: .8rem;
149 text-transform: uppercase; letter-spacing: .04em; }
150 td code { font-size: .82rem; }
151 .scroll { overflow-x: auto; }
152
153 footer {
154 margin-top: 3.5rem; padding-top: 1.5rem;
155 border-top: 1px solid var(--line);
156 font-size: .87rem; color: var(--muted);
157 }
158 footer a { color: var(--accent); }
159 .spin { color: var(--muted); font-size: .9rem; }
160 ol.checks { padding-left: 1.25rem; }
161 ol.checks li { margin: .4rem 0; }
162</style>
163</head>
164<body>
165<div class="wrap">
166
167<header>
168 <a class="brand" href="https://assert.gr" target="_blank" rel="noopener">
169 <img src="./assert-logo.png" alt="ASSERT" width="420" height="87">
170 </a>
171 <h1>Does this pack keep what it claims?</h1>
172 <p>
173 Quantization can drop or disable part of a model without failing. The pack
174 loads, serves, and answers correctly — while a component its card says it
175 preserved is either gone, or present and ignored by the runtime. Nothing
176 errors. Nothing in the logs says so.
177 </p>
178 <p>
179 This reads a published repo's <strong>metadata only</strong>: safetensors
180 headers and <code>config.json</code>, fetched over ranged requests, from
181 your browser. No weights are downloaded, so a 27 GB pack costs a few
182 kilobytes to check.
183 </p>
184</header>
185
186<form id="form">
187 <input type="text" id="repo" value="aleada/Nemotron-3.5-Lightning-30B-A3B-W4A16"
188 placeholder="owner/model" autocomplete="off" spellcheck="false"
189 aria-label="Model repo id">
190 <button type="submit" id="go">Check</button>
191</form>
192<div class="examples">
193 <span class="group">Our packs:</span>
194 <button type="button" data-ex="aleada/Nemotron-3.5-Lightning-30B-A3B-W4A16">Nemotron-3.5-30B</button> ·
195 <button type="button" data-ex="aleada/Qwen3.8-27B-W4A16">Qwen3.8-27B</button> ·
196 <button type="button" data-ex="aleada/Qwen3.6-27B-W4A16">Qwen3.6-27B</button> ·
197 <button type="button" data-ex="aleada/Gemma-4-12B-it-W4A16">Gemma-4-12B</button> ·
198 <button type="button" data-ex="aleada/Pixtral-12B-W4A16">Pixtral-12B</button> ·
199 <button type="button" data-ex="aleada/Llama-3.2-11B-Vision-Instruct-W4A16">Llama-3.2-Vision</button>
200 <br>
201 <span class="group">Elsewhere:</span>
202 <button type="button" data-ex="groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit">one missing its draft head</button> ·
203 <button type="button" data-ex="QuantTrio/Qwen3.6-27B-AWQ">a clean pack, another format</button>
204</div>
205
206<div id="out"></div>
207
208<h2>What it checks</h2>
209<ol class="checks">
210 <li>
211 <strong>Does the index promise weights the repo does not have?</strong>
212 <code>model.safetensors.index.json</code> maps every tensor to a shard and
213 a loader resolves weights through it, so an entry pointing at a file that
214 is not there makes the pack unloadable. This is the one check that holds
215 for any architecture and any quantizer.
216 </li>
217 <li>
218 <strong>Does every exclusion entry name a real module?</strong> An entry
219 naming something this architecture does not have protects nothing, and
220 whatever it was meant to keep was quantized. The common case is a vision
221 tower: Llama and Pixtral call it <code>vision_tower</code>, Qwen calls it
222 <code>visual</code>, and the documented default names the former.
223 </li>
224 <li>
225 <strong>Did any module of the source model fail to reach the pack?</strong>
226 <code>from_pretrained</code> does not materialise tensors the AutoModel
227 class has no slot for, so an auxiliary head living inside the checkpoint
228 never reaches the quantizer and never reaches the artifact.
229 </li>
230 <li>
231 <strong>Is anything at source precision without being declared?</strong> A
232 runtime builds those quantized, looks for a packed weight, finds a plain
233 one, and skips it — leaving the module randomly initialised.
234 </li>
235</ol>
236<p>
237 The first two are observations. The third is a <strong>prediction</strong>,
238 and says so where it appears.
239</p>
240
241<h2>Why the rules are per format</h2>
242<p>
243 Every format declares exclusions differently, and reading one convention into
244 another is how a checker invents verdicts. A bare <code>mtp</code> covers a
245 whole head under AWQ's substring rule and covers <em>nothing</em> under
246 compressed-tensors' exact-match rule.
247</p>
248<div class="scroll">
249<table>
250 <thead><tr><th>Format</th><th>Field</th><th>How an entry matches</th></tr></thead>
251 <tbody>
252 <tr><td>compressed-tensors</td><td><code>ignore</code></td><td>exact equality, or a <code>re:</code> regex</td></tr>
253 <tr><td>awq</td><td><code>modules_to_not_convert</code></td><td>plain substring</td></tr>
254 <tr><td>gptq</td><td><code>dynamic</code></td><td>ordered, first match wins; <code>-:</code> excludes</td></tr>
255 <tr><td>bitsandbytes</td><td><code>llm_int8_skip_modules</code></td><td>exact path component, or a cumulative prefix</td></tr>
256 <tr><td>modelopt</td><td><code>exclude_modules</code></td><td>exact, substring, or an fnmatch glob</td></tr>
257 <tr><td>auto-round</td><td><code>extra_config</code></td><td>per-module <code>bits ≥ 16</code>, or a block allowlist</td></tr>
258 </tbody>
259</table>
260</div>
261<p>
262 Every regex above resolves through Python's <code>re.match</code>, which
263 anchors at the start: <code>re:layers.0</code> does <em>not</em> match
264 <code>mtp.layers.0…</code>. These were read out of vLLM's source, not from
265 documentation. A format not on the list is reported as <strong>not
266 checked</strong> rather than assumed healthy — a checker that cannot tell
267 "healthy" from "not checked" is not a checker.
268</p>
269
270<footer>
271 <p>
272 <strong>A clean result is not a quality measurement.</strong> It means the
273 three checks found nothing. This never loads the model and never runs it. A
274 pack can pass here and still be a poor quantization.
275 </p>
276 <p>
277 Rules extracted from vLLM <code>0.23.1rc1.dev552+g4559c43a9</code> on
278 2026-08-11; they are not a stable public API upstream. Check 3 tolerates
279 the wrapper-level difference between the names written into an exclusion
280 list and the names tensors are stored under, because vLLM resolves that with
281 a per-architecture mapper this cannot replicate from metadata — so it errs
282 toward "declared", missing some real cases rather than accusing correct
283 packs. Gated and private repos cannot be read.
284 </p>
285 <p>
286 Built from the tooling behind the
287 <a href="https://huggingface.co/aleada" rel="noopener">aleada</a> W4A16 packs.
288 See also the <a href="https://huggingface.co/spaces/aleada/pack-precision-map" rel="noopener">precision map</a>, the <a href="https://huggingface.co/spaces/aleada/reasoning-parser-advisor" rel="noopener">reasoning-parser advisor</a> and the <a href="https://huggingface.co/spaces/aleada/model-search-that-answers" rel="noopener">model search</a>.
289 </p>
290</footer>
291
292</div>
293<script type="module" src="./app.js"></script>
294</body>
295</html>
296