agentionai/quant-fidelity-corpora
Quantization fidelity corpora Evaluation text for measuring how faithfully a quantized LLM reproduces its full-precision parent (KL divergence of next-token distributions, top-1 agreement, perplexity ratio), as used by Agention for the Signal and Qwen3.8-27B quantization campaigns. mixedweb-v1 mixedweb-v1.txt (800,789 chars, 301 documents, md5 51e0045e8cabf37922aa82766a25b7b4) is a seeded random slice of HuggingFaceFW/fineweb sample-10BT: general English web text… See the full description on the dataset page: https://huggingface.co/datasets/agentionai/quant-fidelity-corpora.
Quantization fidelity corpora
Evaluation text for measuring how faithfully a quantized LLM reproduces its full-precision parent (KL divergence of next-token distributions, top-1 agreement, perplexity ratio), as used by Agention for the Signal and Qwen3.8-27B quantization campaigns.
mixedweb-v1
mixedweb-v1.txt (800,789 chars, 301 documents, md5 51e0045e8cabf37922aa82766a25b7b4) is a seeded random slice of [HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) `sample-10BT`: general English web text from 55 CommonCrawl dumps (2013-2023). Documents are kept whole up to 6,000 characters (cut at a paragraph boundary), documents under 600 characters are skipped, and they are concatenated with blank lines. mixedweb-v1.manifest.jsonl records the row offset, FineWeb id, URL, dump and date of every document, in order. build_mixedweb_corpus.py is the builder (seed 20260919, datasets-server rows API).
Why a third corpus
Everyone reports wikitext-2. Wikipedia is in every pretraining mix and wikitext is the default calibration text for many quantizers, so a low wikitext KLD can mean "calibrated on Wikipedia" as much as "faithful". A private held-out corpus in the model's real domain fixes that, but is open to the mirror-image objection that it shares a domain with the packager's own calibration set. mixedweb-v1 is the neutral leg: a random slice of a 15M-document web sample that no packager calibrates on. It has zero 12-gram overlap with our calibration set, with our held-out corpus and with wikitext-2.
Our win criterion for a quantization is: better on the held-out corpus AND on mixedweb-v1, with wikitext-2 reported for comparability only.
How we measure
llama-perplexity -m <bf16-reference>.gguf -f mixedweb-v1.txt --kl-divergence-base base.bin -c 2048 --chunks 60 -ngl 99
llama-perplexity -m <quant>.gguf --kl-divergence-base base.bin --kl-divergence -c 2048 -ngl 99The base file fixes the tokenization and chunking, so every quant of the same parent is scored on exactly the same 60 x 2048 tokens. Report Mean KLD with its standard error, Same top p, and Mean PPL(Q).
License
The text is redistributed from FineWeb under the Open Data Commons Attribution License (ODC-By) v1.0; CommonCrawl's terms of use apply to the underlying crawl. The builder script is MIT.