Modelpublic

ENOSYS/Octen-Embedding-0.6B-750-v1-GGUF

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

3likes541downloads

Model Card

Experimental global target bits‑per‑weight quantization of Octen/Octen-Embedding-0.6B

—Using non-standard (forked) LLaMA C++ branch for quantization.
—Using a CLI tool to build KLD evaluation and imatrix calibration datasets for GGUF models, sourced from eaddario/imatrix-calibration.
—Using dataset sources: texten, textru.
—Using dataset chunks: 750.
—Small set of patches added.
—Tensors quantinization F16 instead of BF16, Nvidia Pascal architecture friendly like P100.
—Small set of patches added.

Many thanks to Ed Addario for an impressive job.

Quantization comparison

BPW/TGS	PPL correlation	PPL mean ratio	ΔPPL	Mean KLD	Maximum KLD	99.9% KLD	Mean Δp	RMS Δp
3.50	85.56%	1.328184 ± 0.006198	113.055840 ± 2.030409	1.738910 ± 0.002968	19.720037	10.812436	-3.331 ± 0.037 %	17.941 ± 0.063 %
4.00	93.58%	1.360601 ± 0.004364	124.223093 ± 1.754530	0.787587 ± 0.001566	19.145452	6.742295	-1.688 ± 0.027 %	12.813 ± 0.052 %
4.50	95.88%	1.235695 ± 0.003178	81.194364 ± 1.267961	0.492418 ± 0.001110	16.026636	5.010898	-1.135 ± 0.022 %	10.330 ± 0.047 %
5.00	97.12%	1.225914 ± 0.002645	77.825006 ± 1.132301	0.323054 ± 0.000773	14.785572	3.622450	-1.015 ± 0.018 %	8.707 ± 0.042 %
5.50	98.61%	1.140659 ± 0.001722	48.455459 ± 0.754229	0.142510 ± 0.000361	10.251470	1.794577	-0.500 ± 0.012 %	5.834 ± 0.031 %
6.00	99.03%	1.095986 ± 0.001385	33.066115 ± 0.583789	0.089186 ± 0.000256	9.821702	1.338731	-0.228 ± 0.010 %	4.639 ± 0.028 %
6.50	99.28%	1.086861 ± 0.001185	29.922486 ± 0.515390	0.056230 ± 0.000156	9.794716	0.742507	-0.200 ± 0.008 %	3.695 ± 0.022 %
7.00	99.47%	1.032955 ± 0.000969	11.352594 ± 0.361050	0.033634 ± 0.000093	3.941931	0.450176	-0.041 ± 0.006 %	2.933 ± 0.020 %
7.50	99.53%	1.015463 ± 0.000891	5.326939 ± 0.316756	0.024763 ± 0.000071	4.088861	0.356147	0.027 ± 0.005 %	2.558 ± 0.019 %
8.00	99.56%	1.012318 ± 0.000866	4.243431 ± 0.305956	0.021680 ± 0.000059	1.800071	0.296283	0.037 ± 0.005 %	2.380 ± 0.015 %
8.50	99.63%	1.020174 ± 0.000803	6.949556 ± 0.292500	0.013173 ± 0.000038	3.675829	0.164360	-0.008 ± 0.004 %	1.903 ± 0.015 %
9.00	99.64%	1.017785 ± 0.000789	6.126855 ± 0.285191	0.011793 ± 0.000035	3.644340	0.155200	0.001 ± 0.004 %	1.822 ± 0.015 %
9.50	99.64%	1.023754 ± 0.000790	8.182888 ± 0.292987	0.011307 ± 0.000036	3.703608	0.149881	-0.017 ± 0.004 %	1.799 ± 0.017 %
10.00	99.64%	1.026870 ± 0.000790	9.256477 ± 0.297126	0.010960 ± 0.000039	4.368244	0.146243	-0.023 ± 0.004 %	1.781 ± 0.017 %
10.50	99.65%	1.031793 ± 0.000791	10.952329 ± 0.305078	0.010631 ± 0.000033	2.633135	0.139757	-0.033 ± 0.004 %	1.756 ± 0.016 %
11.00	99.65%	1.032131 ± 0.000785	11.068814 ± 0.303826	0.010088 ± 0.000026	1.066854	0.125272	-0.039 ± 0.004 %	1.698 ± 0.010 %
11.50	99.66%	1.034359 ± 0.000785	11.836127 ± 0.307385	0.009816 ± 0.000026	1.311119	0.122771	-0.039 ± 0.004 %	1.685 ± 0.010 %
12.00	99.66%	1.033216 ± 0.000782	11.442555 ± 0.304820	0.009535 ± 0.000028	2.683337	0.118959	-0.036 ± 0.004 %	1.669 ± 0.015 %
12.50	99.66%	1.035454 ± 0.000782	12.213620 ± 0.308823	0.009296 ± 0.000026	1.888097	0.117823	-0.036 ± 0.003 %	1.653 ± 0.011 %
13.00	99.66%	1.035315 ± 0.000779	12.165472 ± 0.307315	0.009015 ± 0.000024	1.563708	0.112765	-0.037 ± 0.003 %	1.641 ± 0.012 %
13.50	99.66%	1.035511 ± 0.000777	12.233185 ± 0.307197	0.008828 ± 0.000027	2.020073	0.110392	-0.042 ± 0.003 %	1.634 ± 0.016 %
14.00	99.67%	1.034812 ± 0.000775	11.992309 ± 0.305653	0.008529 ± 0.000025	1.908343	0.105182	-0.034 ± 0.003 %	1.617 ± 0.017 %
14.50	99.66%	1.035023 ± 0.000780	12.064932 ± 0.307359	0.008970 ± 0.000023	0.820967	0.111618	-0.034 ± 0.003 %	1.618 ± 0.010 %
15.00	99.66%	1.035123 ± 0.000777	12.099435 ± 0.306609	0.008687 ± 0.000022	1.018949	0.102148	-0.033 ± 0.003 %	1.612 ± 0.010 %