GeekedOutAi/Geeked-Out-Quantization-Software

1# Quantization Notes2 3## Overview4 5This model was quantized using **The Geeked Out Quantizer**, a specialized Windows-native quantization environment designed for extreme compression with quality preservation.6 7## Quantization Specifications8 9| Parameter | Details |10|-----------|---------|11| **Source Format** | BF16 (bfloat16) or F16 (float16) |12| **Target Format** | IQ2_M (2.0 bits per weight) |13| **Compression Ratio** | 16x smaller than FP32 baseline |14| **Quantization Method** | Importance-aware quantization with IMatrix |15| **Quality Metric** | ~3-8% perplexity increase vs. baseline |16 17## The Importance Matrix (IMatrix) Method18 19### What is an Importance Matrix?20 21An importance matrix is a statistical analysis of a neural network that identifies which weights contribute most significantly to model output quality. Rather than applying uniform quantization across all tensors, this method:22 23- **Preserves precision** on high-impact weights24- **Aggressively compresses** low-impact weights25- **Maintains information flow** through the network architecture26 27### Why It Matters28 29Traditional uniform quantization to 2-bit precision typically causes 10-20% quality degradation. The importance matrix approach reduces this to 3-8%, making 2-bit models viable for production use.30 31## Calibration Process32 33### Data Selection34 35The importance matrix is generated using carefully selected calibration data that:36- Represents the model's intended use domain37- Contains diverse vocabulary and sentence structures38- Includes 100-500 text chunks of typical prompt length39- Matches the distribution of expected inference inputs40 41### Generation Parameters42 43| Setting | Typical Value | Purpose |44|---------|---------------|---------|45| Chunks | 200-500 | Balance quality vs. generation time |46| GPU Layers | 99 (max) | Accelerate processing via CUDA |47| Thread Count | Auto-detected | Optimize for hardware configuration |48 49## Memory & Hardware Optimization50 51The quantization process includes:52- **Dynamic memory management** — Reserves system RAM to maintain Windows responsiveness53- **Hardware detection** — Automatically detects CPU cores, memory type (DDR4/DDR5), and GPU capabilities54- **Thread optimization** — Adjusts parallelism based on available resources55- **Retry logic** — Handles transient memory pressure gracefully56 57## Model Selection Criteria58 59Source models are selected based on quality hierarchy:601. **BF16** (preferred) — Best precision for quantization612. **F16** — Good precision, widely available623. **F32** — Acceptable but creates larger intermediate files63 64Models already in quantized formats are skipped unless explicitly re-quantizing.65 66## Output Format Details67 68### IQ2_M Characteristics69 70- **Bit depth:** 2.0 bits per weight71- **Speed:** 2-3x faster inference than F3272- **VRAM usage:** ~1/16th of FP3273- **Imatrix required:** Yes74- **Quality tier:** Best-in-class for 2-bit quantization75 76### Naming Convention77 78Quantized models follow this pattern:79```80OriginalModel-BF16.gguf → OriginalModel-IQ2_M.gguf81```82 83Sharded models preserve shard numbering:84```85Model-00001-of-00004.gguf → Model-IQ2_M-00001-of-00004.gguf86```87 88## Quality Verification89 90Models are validated through:91- Perplexity measurement against baseline92- Sample inference testing93- File integrity verification94 95## Use Cases96 97IQ2_M quantized models are ideal for:98- **Edge deployment** — Minimal storage footprint99- **Consumer hardware** — Reduced VRAM requirements100- **High-throughput inference** — Faster token generation101- **Bandwidth-constrained environments** — Efficient distribution102 103## Technical Notes104 105- Quantization performed on Windows with CUDA 12.4+ support106- GPU acceleration utilized for imatrix generation107- Multi-threaded processing with memory safety guards108- Compatible with llama.cpp inference engines109 110## Citation111 112If you use this quantized model in research or applications, please acknowledge:113 114> Quantized using The Geeked Out Quantizer with importance-aware IQ2_M optimization.115 116---117 118*For questions about the quantization method or collaboration inquiries, please open a discussion on this model's page.*119