CoolFace
Modelpublic

ENOSYS/GigaChat3.1-10B-A1.8B-750-v1-GGUF

sourceHugging Facemitupdated 5mo agoView on Hugging Face
3likes3.2kdownloads
Model Card

Experimental global target bits‑per‑weight quantization of ai-sage/GigaChat3.1-10B-A1.8B-bf16

  • —Using non-standard (forked) LLaMA C++ branch for quantization.
  • —Using a CLI tool to build KLD evaluation and imatrix calibration datasets for GGUF models, sourced from eaddario/imatrix-calibration.
  • —Using dataset sources: tools, math, code, texten, textru.
  • —Using dataset chunks: 750.
  • —Tensors quantinization F16 instead of BF16, Nvidia Pascal architecture friendly like P100.
  • —Small set of patches added.

Many thanks to Ed Addario for an impressive job.

Quantization comparison

BPW/TGSSIZE (MiB)PPL correlationPPL mean ratioΔPPLMean KLDMedian KLDMaximum KLD99.9% KLDMean ΔpRMS Δp
2.750350490.09%1.468708 ± 0.0055217.175448 ± 0.0989130.633578 ± 0.0023630.31392630.66096710.392284-3.925 ± 0.046 %21.062 ± 0.068 %
2.875366390.85%1.367633 ± 0.0048485.628103 ± 0.0825930.562811 ± 0.0021700.27734125.64353410.194194-3.832 ± 0.044 %20.141 ± 0.067 %
3.000382291.16%1.358840 ± 0.0047505.493483 ± 0.0814730.538277 ± 0.0020950.26396727.6319039.956902-3.381 ± 0.043 %19.653 ± 0.067 %
3.125398193.15%1.238643 ± 0.0037733.653381 ± 0.0623620.421670 ± 0.0018000.20037128.3513729.239901-2.627 ± 0.038 %17.619 ± 0.064 %
3.250414093.72%1.236406 ± 0.0036153.619143 ± 0.0606360.385529 ± 0.0017040.18370928.9033018.885584-3.066 ± 0.037 %17.138 ± 0.064 %
3.375429994.49%1.182225 ± 0.0032132.789691 ± 0.0523340.335596 ± 0.0015370.16164226.3399418.339195-2.754 ± 0.035 %16.065 ± 0.062 %
3.500445895.70%1.098661 ± 0.0026121.510403 ± 0.0405580.242724 ± 0.0013340.09524530.3487007.182995-2.212 ± 0.031 %14.103 ± 0.063 %
3.625461795.60%1.077755 ± 0.0025641.190348 ± 0.0387730.254184 ± 0.0012990.10724834.7465636.869932-1.966 ± 0.031 %14.120 ± 0.061 %
3.750477696.46%1.059019 ± 0.0022780.903527 ± 0.0348630.199596 ± 0.0011360.07960628.7987386.495603-1.650 ± 0.028 %12.762 ± 0.060 %
3.875493596.56%1.014364 ± 0.0021300.219904 ± 0.0323250.193066 ± 0.0010970.08099528.8502016.062064-1.619 ± 0.027 %12.511 ± 0.058 %
4.000509496.14%1.040658 ± 0.0023170.622426 ± 0.0349360.212281 ± 0.0011070.09506734.2688185.975020-0.839 ± 0.028 %12.849 ± 0.057 %
4.125525397.13%0.989743 ± 0.001899-0.157030 ± 0.0293190.158050 ± 0.0009850.06671533.0860755.331661-1.389 ± 0.025 %11.412 ± 0.056 %
4.250541297.56%0.991427 ± 0.001760-0.131246 ± 0.0270990.128950 ± 0.0008830.05314733.5787204.832679-0.858 ± 0.023 %10.258 ± 0.054 %
4.375557198.42%1.014575 ± 0.0014600.223132 ± 0.0223680.080255 ± 0.0006760.02719425.9350363.807253-0.339 ± 0.018 %8.266 ± 0.053 %
4.500573098.53%1.020259 ± 0.0014240.310144 ± 0.0219330.074827 ± 0.0007090.02478228.5918463.737915-0.211 ± 0.017 %7.880 ± 0.052 %
4.625589098.43%1.012727 ± 0.0014560.194840 ± 0.0223050.077205 ± 0.0006750.02573325.9615363.821413-0.249 ± 0.018 %8.150 ± 0.053 %
4.750604998.69%1.005256 ± 0.0013180.080470 ± 0.0201710.065456 ± 0.0006110.02182823.6095433.366829-0.227 ± 0.017 %7.487 ± 0.052 %
4.875620898.68%1.007570 ± 0.0013290.115888 ± 0.0203550.064722 ± 0.0006220.02202723.7764113.370074-0.192 ± 0.016 %7.400 ± 0.051 %
5.000636798.76%1.003428 ± 0.0012790.052475 ± 0.0195720.060466 ± 0.0005790.02074725.8111503.064398-0.248 ± 0.016 %7.209 ± 0.051 %
5.125652698.77%1.006995 ± 0.0012820.107087 ± 0.0196320.059681 ± 0.0005990.02044832.1015282.992807-0.140 ± 0.016 %7.118 ± 0.050 %
5.250668598.83%1.002481 ± 0.0012430.037985 ± 0.0190230.056570 ± 0.0006090.01917433.0789152.996349-0.198 ± 0.015 %6.909 ± 0.050 %
5.375684498.89%1.006384 ± 0.0012160.097726 ± 0.0186240.053188 ± 0.0005810.01763429.2202492.952291-0.157 ± 0.015 %6.679 ± 0.050 %
5.500700398.98%1.011065 ± 0.0011770.169390 ± 0.0181050.048359 ± 0.0005250.01586824.7845342.4871710.002 ± 0.014 %6.533 ± 0.049 %
5.625716299.18%0.993691 ± 0.001026-0.096579 ± 0.0157380.035009 ± 0.0005020.01006524.7829362.058697-0.135 ± 0.012 %5.578 ± 0.050 %
5.750732199.21%1.001571 ± 0.0010220.024053 ± 0.0156560.032926 ± 0.0004780.00923628.8909031.993644-0.053 ± 0.012 %5.388 ± 0.049 %
5.875748099.28%1.000869 ± 0.0009740.013302 ± 0.0149160.029095 ± 0.0004090.00816322.8795321.781184-0.064 ± 0.011 %5.062 ± 0.047 %
6.000763999.32%1.008715 ± 0.0009560.133418 ± 0.0147360.027239 ± 0.0004370.00751122.6812361.728597-0.083 ± 0.011 %4.871 ± 0.048 %
6.125779899.34%1.002073 ± 0.0009390.031741 ± 0.0143790.025381 ± 0.0003720.00735422.0273901.516017-0.093 ± 0.010 %4.741 ± 0.046 %
6.250795799.37%1.020782 ± 0.0009450.318146 ± 0.0149720.024460 ± 0.0004320.00662725.3938431.640636-0.052 ± 0.010 %4.564 ± 0.047 %
6.375811699.39%1.012039 ± 0.0009190.184303 ± 0.0142820.023308 ± 0.0004160.00622823.9820161.526792-0.043 ± 0.010 %4.502 ± 0.047 %
6.500827599.40%1.020302 ± 0.0009180.310798 ± 0.0145360.022949 ± 0.0004110.00606125.4412561.560629-0.073 ± 0.010 %4.459 ± 0.047 %
6.625843499.50%1.015903 ± 0.0008360.243456 ± 0.0131480.017084 ± 0.0003490.00440421.8994011.088672-0.047 ± 0.009 %3.880 ± 0.046 %
6.750859399.54%1.025129 ± 0.0008190.384694 ± 0.0133810.014613 ± 0.0003240.00334622.1349051.0440710.025 ± 0.008 %3.586 ± 0.047 %
6.875875299.53%1.021068 ± 0.0008190.322524 ± 0.0131520.014827 ± 0.0003450.00337431.3169541.0476630.000 ± 0.008 %3.596 ± 0.046 %
7.000891199.55%1.024709 ± 0.0008050.378267 ± 0.0131540.014163 ± 0.0003070.00324521.2719841.0474680.011 ± 0.008 %3.549 ± 0.047 %
7.125907099.55%1.022398 ± 0.0008070.342897 ± 0.0130450.014525 ± 0.0003270.00329521.5034291.0244290.014 ± 0.008 %3.593 ± 0.048 %
7.250922999.56%1.028127 ± 0.0008080.430596 ± 0.0134120.013705 ± 0.0002890.00314719.2225001.0207730.032 ± 0.008 %3.505 ± 0.047 %
7.375938899.57%1.027008 ± 0.0007920.413460 ± 0.0131270.013062 ± 0.0002780.00288018.6018680.9915050.021 ± 0.008 %3.478 ± 0.049 %
7.500954799.58%1.022934 ± 0.0007820.351091 ± 0.0127300.012368 ± 0.0002910.00272918.9610080.9003890.011 ± 0.007 %3.332 ± 0.047 %
7.625970699.57%1.027221 ± 0.0007930.416723 ± 0.0131550.012728 ± 0.0003340.00282126.7436620.8899150.039 ± 0.007 %3.399 ± 0.048 %
7.750986599.58%1.025006 ± 0.0007840.382820 ± 0.0128890.012181 ± 0.0002860.00268917.7618310.9252880.023 ± 0.007 %3.367 ± 0.048 %
7.8751002499.58%1.026968 ± 0.0007850.412852 ± 0.0130410.012134 ± 0.0002970.00271720.4705410.8584580.043 ± 0.007 %3.291 ± 0.045 %
8.0001018399.59%1.024610 ± 0.0007770.376758 ± 0.0127600.011861 ± 0.0003020.00260023.9169350.8929560.008 ± 0.007 %3.292 ± 0.048 %
8.1251034299.60%1.025372 ± 0.0007630.388413 ± 0.0126230.011646 ± 0.0002800.00263619.3655340.8550190.028 ± 0.007 %3.250 ± 0.045 %
8.2501050199.60%1.024733 ± 0.0007650.378644 ± 0.0126060.011539 ± 0.0002900.00250523.3506410.8383220.004 ± 0.007 %3.250 ± 0.047 %
8.3751066199.60%1.022929 ± 0.0007610.351025 ± 0.0124420.011353 ± 0.0003030.00242924.3818650.8405600.023 ± 0.007 %3.225 ± 0.048 %
8.5001081999.61%1.021716 ± 0.0007520.332456 ± 0.0122460.010923 ± 0.0002780.00237525.4040780.8043610.025 ± 0.007 %3.175 ± 0.049 %
8.6251097899.61%1.023591 ± 0.0007500.361155 ± 0.0123220.010546 ± 0.0002770.00215118.9151780.8214810.022 ± 0.007 %3.139 ± 0.050 %
8.7501113799.66%1.016991 ± 0.0006960.260111 ± 0.0111900.008489 ± 0.0002640.00157922.0720560.7058300.024 ± 0.006 %2.832 ± 0.052 %
8.8751129699.66%1.016703 ± 0.0006910.255707 ± 0.0111000.008476 ± 0.0002770.00156220.4737990.6836930.014 ± 0.006 %2.759 ± 0.050 %
9.0001145699.66%1.016588 ± 0.0006950.253941 ± 0.0111520.008375 ± 0.0002740.00142220.3146760.6845100.018 ± 0.006 %2.763 ± 0.050 %
9.1251161499.66%1.017290 ± 0.0006920.264687 ± 0.0111470.008240 ± 0.0002740.00138420.7994900.6587570.029 ± 0.006 %2.735 ± 0.050 %
9.2501177499.66%1.018057 ± 0.0006970.276428 ± 0.0112690.008108 ± 0.0002280.00140316.4911000.7117420.035 ± 0.006 %2.749 ± 0.048 %
9.3751193299.67%1.015225 ± 0.0006850.233083 ± 0.0109370.007767 ± 0.0002700.00128919.4639870.6748940.012 ± 0.006 %2.675 ± 0.048 %
9.5001209199.67%1.018677 ± 0.0006920.285927 ± 0.0112230.008071 ± 0.0002720.00135422.1778760.6908200.034 ± 0.006 %2.780 ± 0.051 %
9.6251225099.67%1.014632 ± 0.0006830.224007 ± 0.0108750.007610 ± 0.0002800.00127624.4191280.6199480.015 ± 0.006 %2.639 ± 0.048 %
9.7501240899.67%1.014877 ± 0.0006840.227755 ± 0.0109130.007592 ± 0.0002620.00123119.5171890.6014380.014 ± 0.006 %2.645 ± 0.050 %
9.8751256899.67%1.013450 ± 0.0006790.205913 ± 0.0107690.007440 ± 0.0002770.00120220.1402680.6365590.022 ± 0.006 %2.612 ± 0.050 %
10.0001272799.67%1.014093 ± 0.0006830.215754 ± 0.0108610.007558 ± 0.0002740.00123119.7142680.6329210.014 ± 0.006 %2.641 ± 0.050 %
10.1251288699.67%1.013744 ± 0.0006780.210406 ± 0.0107620.007376 ± 0.0002460.00118619.4427030.6309770.010 ± 0.006 %2.646 ± 0.053 %
10.2501304699.67%1.013908 ± 0.0006800.212922 ± 0.0108080.007539 ± 0.0002810.00123123.7975900.6071420.013 ± 0.006 %2.639 ± 0.050 %
10.3751320499.68%1.014028 ± 0.0006730.214749 ± 0.0107140.007474 ± 0.0002720.00119320.7188990.6122450.013 ± 0.006 %2.612 ± 0.050 %
10.5001336499.67%1.013952 ± 0.0006760.213597 ± 0.0107500.007299 ± 0.0002610.00114519.4104800.6187630.013 ± 0.006 %2.578 ± 0.049 %
10.6251352399.67%1.013976 ± 0.0006770.213963 ± 0.0107630.007451 ± 0.0002610.00118920.5692730.6224270.019 ± 0.006 %2.592 ± 0.049 %
10.7501368099.67%1.014306 ± 0.0006770.219003 ± 0.0107820.007248 ± 0.0002470.00114819.2282980.6132740.011 ± 0.006 %2.586 ± 0.050 %
10.8751384099.67%1.015255 ± 0.0006790.233540 ± 0.0108560.007507 ± 0.0002810.00119320.1994930.5938700.017 ± 0.006 %2.613 ± 0.050 %
11.0001399999.67%1.014683 ± 0.0006780.224779 ± 0.0108160.007216 ± 0.0002470.00114517.9416960.6470050.016 ± 0.006 %2.611 ± 0.051 %
11.1251415999.69%1.013409 ± 0.0006640.205278 ± 0.0105430.007003 ± 0.0002580.00108318.8742430.5920080.017 ± 0.006 %2.540 ± 0.048 %
11.2501431899.69%1.012913 ± 0.0006640.197686 ± 0.0105200.006675 ± 0.0002450.00105319.1686440.5632730.015 ± 0.006 %2.505 ± 0.050 %
11.3751447799.68%1.013141 ± 0.0006680.201168 ± 0.0105950.006782 ± 0.0002370.00108718.6734680.5866650.015 ± 0.005 %2.490 ± 0.048 %
11.5001463699.69%1.012523 ± 0.0006600.191714 ± 0.0104540.006529 ± 0.0002360.00104218.6471630.5483470.019 ± 0.005 %2.449 ± 0.049 %
11.6251479299.69%1.013233 ± 0.0006570.202580 ± 0.0104380.006673 ± 0.0002540.00102718.9591240.5652850.015 ± 0.005 %2.463 ± 0.051 %
11.7501495399.68%1.013257 ± 0.0006690.202953 ± 0.0106130.007152 ± 0.0002850.00104718.9186000.6302950.018 ± 0.006 %2.532 ± 0.051 %
11.8751511399.69%1.013355 ± 0.0006610.204458 ± 0.0104930.006514 ± 0.0002310.00101319.0428580.5754340.011 ± 0.005 %2.461 ± 0.048 %
12.0001527199.68%1.013285 ± 0.0006650.203384 ± 0.0105510.006936 ± 0.0002680.00106521.0288120.5920600.013 ± 0.006 %2.541 ± 0.051 %
12.1251543199.69%1.013224 ± 0.0006600.202447 ± 0.0104880.006615 ± 0.0002350.00102719.0681920.6027030.024 ± 0.006 %2.530 ± 0.051 %
12.2501558899.69%1.013507 ± 0.0006590.206774 ± 0.0104760.006742 ± 0.0002700.00100519.1977040.5607960.016 ± 0.005 %2.460 ± 0.049 %
12.3751574999.68%1.012585 ± 0.0006660.192658 ± 0.0105490.006849 ± 0.0002630.00103219.0466140.6243020.023 ± 0.006 %2.511 ± 0.050 %
12.5001590999.69%1.013416 ± 0.0006560.205388 ± 0.0104260.006976 ± 0.0003410.00099630.0707820.5531840.015 ± 0.005 %2.453 ± 0.051 %
12.6251606899.69%1.013417 ± 0.0006640.205405 ± 0.0105510.006847 ± 0.0002550.00103418.8726830.5929050.021 ± 0.006 %2.511 ± 0.051 %
12.7501622599.69%1.013988 ± 0.0006610.214149 ± 0.0105240.006546 ± 0.0002550.00099719.9836790.5709020.020 ± 0.005 %2.420 ± 0.047 %
12.8751638499.69%1.012895 ± 0.0006560.197416 ± 0.0104050.006727 ± 0.0002760.00097720.8823620.5482200.020 ± 0.005 %2.481 ± 0.052 %
13.0001654499.68%1.012696 ± 0.0006680.194360 ± 0.0105720.006706 ± 0.0002730.00099721.3688700.5776130.015 ± 0.005 %2.482 ± 0.051 %
13.1251669699.69%1.012827 ± 0.0006550.196367 ± 0.0103960.006580 ± 0.0002760.00096119.6363490.5392850.022 ± 0.005 %2.418 ± 0.049 %
13.2501686299.70%1.013051 ± 0.0006520.199793 ± 0.0103560.006516 ± 0.0002450.00101118.3785950.5849260.018 ± 0.005 %2.416 ± 0.049 %
13.3751702299.70%1.012315 ± 0.0006500.188526 ± 0.0102980.006461 ± 0.0002710.00096719.5419450.5321160.014 ± 0.005 %2.427 ± 0.050 %
13.5001718099.69%1.012152 ± 0.0006610.186035 ± 0.0104460.006609 ± 0.0002560.00100119.8192750.5793430.012 ± 0.005 %2.414 ± 0.050 %
13.6251734099.69%1.012344 ± 0.0006630.188980 ± 0.0104780.006545 ± 0.0002660.00096721.7221150.5815320.014 ± 0.005 %2.406 ± 0.049 %
13.7501749299.69%1.012725 ± 0.0006560.194806 ± 0.0103990.006602 ± 0.0003110.00094731.3620300.5585760.016 ± 0.005 %2.458 ± 0.052 %
13.8751765799.69%1.012659 ± 0.0006540.193798 ± 0.0103720.006449 ± 0.0002400.00098018.9289630.5847100.017 ± 0.005 %2.418 ± 0.049 %
14.0001781399.70%1.012146 ± 0.0006510.185948 ± 0.0103070.006304 ± 0.0002350.00094818.7586670.5372410.018 ± 0.005 %2.407 ± 0.050 %
14.1251797599.68%1.013601 ± 0.0006670.208224 ± 0.0105930.006542 ± 0.0002460.00097621.4164830.6333200.009 ± 0.005 %2.474 ± 0.053 %
14.2501813499.69%1.012774 ± 0.0006550.195555 ± 0.0103830.006236 ± 0.0002470.00093418.7889880.5385960.012 ± 0.005 %2.376 ± 0.048 %
14.3751828899.70%1.012470 ± 0.0006490.190906 ± 0.0102820.006284 ± 0.0002960.00090631.8029670.5472450.022 ± 0.005 %2.411 ± 0.052 %
14.5001845399.70%1.012954 ± 0.0006530.198314 ± 0.0103670.006192 ± 0.0002300.00094518.9922790.5746980.018 ± 0.005 %2.371 ± 0.047 %
14.6251860999.70%1.013029 ± 0.0006520.199464 ± 0.0103500.006264 ± 0.0002970.00090329.0765590.5329400.017 ± 0.005 %2.400 ± 0.052 %
14.7501877199.70%1.012923 ± 0.0006500.197831 ± 0.0103280.006385 ± 0.0002570.00095119.1785850.5405400.023 ± 0.005 %2.390 ± 0.050 %
14.8751892999.69%1.012335 ± 0.0006570.188844 ± 0.0104020.006149 ± 0.0002330.00092017.7336600.5233300.012 ± 0.005 %2.344 ± 0.047 %
15.0001908499.70%1.012698 ± 0.0006490.194390 ± 0.0103000.006208 ± 0.0002920.00089129.1461890.5309630.019 ± 0.005 %2.389 ± 0.051 %
15.1251924799.69%1.012992 ± 0.0006550.198896 ± 0.0104070.006141 ± 0.0002450.00091019.3241580.5479460.019 ± 0.005 %2.343 ± 0.048 %
15.2501940699.70%1.013381 ± 0.0006460.204855 ± 0.0102860.006350 ± 0.0002570.00088717.9330920.5922170.023 ± 0.005 %2.422 ± 0.051 %
15.3751956499.70%1.012699 ± 0.0006500.194414 ± 0.0103150.006196 ± 0.0002850.00086124.9183960.5197310.024 ± 0.005 %2.337 ± 0.051 %
15.5001971999.70%1.012664 ± 0.0006520.193868 ± 0.0103420.005820 ± 0.0002370.00083818.3797490.4926150.028 ± 0.005 %2.299 ± 0.050 %
15.6251988499.70%1.013846 ± 0.0006510.211964 ± 0.0103860.006172 ± 0.0002780.00086726.0331940.5051560.023 ± 0.005 %2.343 ± 0.050 %
15.7502003999.70%1.013008 ± 0.0006510.199144 ± 0.0103520.005814 ± 0.0002380.00083618.4260230.4925680.023 ± 0.005 %2.297 ± 0.050 %
15.8752020299.70%1.013150 ± 0.0006500.201306 ± 0.0103310.005794 ± 0.0002360.00082418.2699390.4953340.023 ± 0.005 %2.287 ± 0.050 %
16.0002035499.71%1.011306 ± 0.0006390.173086 ± 0.0100840.005484 ± 0.0002440.00075018.4949050.4774120.018 ± 0.005 %2.187 ± 0.049 %