GeekedOutAi/Geeked-Out-Quantization-Software

1CALIBRATION DATA INFORMATION2=============================3 4This model was quantized using importance matrix (imatrix) generation.5The imatrix captures which weights in the model are most important for6maintaining output quality during extreme compression (2-bit quantization).7 8WHAT IS CALIBRATION?9--------------------10Calibration is the process of running sample inputs through the model to11measure which tensors (weight matrices) contribute most to the output.12These measurements create an "importance matrix" that guides the quantizer13to preserve precision where it matters most.14 15CALIBRATION DATA CHARACTERISTICS16--------------------------------17Good calibration data should be:18 191. REPRESENTATIVE20   - Matches the domain the model will operate in21   - Similar vocabulary and complexity to expected inputs22   - Reflects actual use case scenarios23 242. DIVERSE25   - Multiple topics, subjects, and writing styles26   - Mix of common and rare tokens27   - Varied sentence structures and lengths28 293. SUFFICIENT30   - 100-500 text chunks of typical document length31   - More chunks = better quality (diminishing returns beyond ~500)32   - Each chunk processed independently33 344. NATURAL35   - Real-world text (not synthetic or random)36   - Domain-appropriate (code for code models, medical for medical models)37   - Representative token distribution38 39CALIBRATION PROCESS PARAMETERS40------------------------------41Typical settings for this quantization:42 43  Chunks Processed:  200-500 (production quality)44  Chunk Size:        Typical document/paragraph length45  GPU Acceleration:  Enabled (99 layers offloaded)46  Thread Count:      Auto-detected based on CPU47 48QUALITY IMPACT49--------------50The importance matrix generated from quality calibration data enables:51 52  - 3-8% perplexity increase (vs 10-20% without imatrix)53  - Preservation of critical weights54  - Intelligent bit allocation per tensor55  - 16x compression with minimal quality loss56 57CALIBRATION DATA SOURCES58------------------------59Common sources for high-quality calibration data:60 61  - Wikitext-2-raw (general language models)62  - Domain-specific corpora (medical, legal, code)63  - The Pile subset (diverse web text)64  - Custom curated datasets matching expected use65 66VERIFICATION67------------68Quantized models are tested for:69  ✓ Perplexity measurement vs baseline70  ✓ Sample inference quality71  ✓ Token prediction accuracy72  ✓ Model file integrity73 74NOTES75-----76- Calibration is performed once per source model77- Same imatrix can be reused for different target formats78- Domain-specific calibration yields better results79- GPU acceleration significantly speeds up generation80 81For questions about the calibration methodology used for this model,82please open a discussion on the model's Hugging Face page.83