CoolFace
Modelpublic

KikoCis/ALIA-40b-instruct-2601-GGUF

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
0likes310downloads
Model Card

[image]

<div style="border:2px solid currentColor; font-family:ui-monospace,'SF Mono','Cascadia Mono',Consolas,monospace;"> <div style="border-bottom:1px solid currentColor; padding:6px 12px; font-size:11px; letter-spacing:3px; text-transform:uppercase; opacity:0.7; text-align:center;">KIKOCIS // EU-SOVEREIGN LLM // IMATRIX GGUF + KLD</div> <div style="padding:14px; display:flex; flex-wrap:wrap; align-items:center; justify-content:center; gap:18px;"> <pre style="margin:0; font-size:9px; line-height:1.15;"> ╔═══════════════════╗ ║ A L I A · 40B ║ ES·CA·GL·EU ╚═══════════════════╝ ┌───┐ ┌───┐ ┌───┐ ┌───┐ │IQ2│ │Q3 │ │Q4 │ │Q8 │ └───┘ └───┘ └───┘ └───┘ ●─────●─────●─────● KLD vs Q8-ref </pre> <div style="text-align:center;"> <div style="font-size:23px; font-weight:800; letter-spacing:1px;">ALIA-40b · GGUF</div> <div style="font-size:12.5px; letter-spacing:1px; opacity:0.8; margin-top:5px;"><span style="white-space:nowrap;">llama · 46B</span> · <span style="white-space:nowrap;">160K ctx</span> · <span style="white-space:nowrap;">imatrix (es)</span> · <span style="white-space:nowrap;">KLD-measured</span></div> </div> </div> <table style="display:table; table-layout:fixed; width:100%; margin:0; border-collapse:collapse; font-size:12px;"> <tr> <td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">FORMAT</div><div style="font-weight:700;">GGUF (imatrix)</div></td> <td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">SIZES</div><div style="font-weight:700;">~13.5 – 40 GB</div></td> <td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">ARCH</div><div style="font-weight:700;">Llama · 46B · 48L</div></td> <td style="border-top:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">CONTEXT</div><div style="font-weight:700;">163840</div></td> </tr> <tr> <td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">IMATRIX</div><div style="font-weight:700;">es corpus</div></td> <td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">VALIDATION</div><div style="font-weight:700;">KLD vs Q8</div></td> <td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">LANGUAGES</div><div style="font-weight:700;">ES·CA·GL·EU·EN</div></td> <td style="border-top:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">LICENSE</div><div style="font-weight:700;">Apache-2.0</div></td> </tr> </table> </div>

ALIA-40b-instruct-2601 — GGUF (imatrix + KLD)

imatrix-quantized GGUFs of ALIA-40b, the Barcelona Supercomputing Center's ~46B sovereign LLM for Spain / the EU (Spanish + Catalan, Galician, Basque + European languages), with a 160K native context. Runs from ~13.5 GB (IQ2M) to **~40 GB** (Q80). Calibrated (imatrix) on Spanish text, with per-quant KLD fidelity vs the near-lossless Q8_0 reference. Credit: this is BSC-LT's model — `BSC-LT/ALIA-40b-instruct-2601`; ours is the quant ladder + metrics.

✅ Recommended files

Use caseFileNotes
Safe defaultALIA-40b-instruct-2601-Q4_K_M.ggufThe one most people want — good quality, ~23 GB (48 GB RAM).
Strong quality/sizeALIA-40b-instruct-2601-Q5_K_M.ggufCloser to the reference, ~27 GB.
Runs on 32 GBALIA-40b-instruct-2601-Q3_K_M.gguf~19 GB — fits a 32 GB machine.
Smallest (tight RAM)ALIA-40b-instruct-2601-IQ2_M.gguf~13.5 GB i-quant — runs a 46B on 24–32 GB, at a real quality cost.
Reference / max fidelityALIA-40b-instruct-2601-Q8_0.gguf~40 GB, near-lossless (the KLD reference).

📦 Files (the ladder)

QuantBitsFile sizeRAM (approx)Notes
IQ2_M~2.7~13.5 GB24–32 GBSmallest — aggressive i-quant.
Q3KM3~18.7 GB32 GBFits a 32 GB machine.
Q4KM4~22.9 GB48 GBSafe default.
Q5KM5~26.8 GB48–64 GBStrong quality/size.
Q8_08~40 GB64 GB+Reference, near-lossless.

<!-- A 46B is big: even IQ2_M needs ~24 GB RAM. Higher number = more bits = closer to the original + bigger. -->

📊 Metrics — fidelity vs the Q8_0 reference

KLD (Kullback–Leibler divergence, nats) measures how far each quant's output distribution drifts from the reference — lower = closer. Top-1 match = how often the quant's top token agrees with the reference. Measured with llama-perplexity --kl-divergence over a Spanish corpus at ctx 512.

Why the reference is Q8_0, not F16? ALIA-40b's F16 is ~81 GB and does not fit this machine's GPU (Metal). Q80 is near-lossless (its own KLD vs F16 would be ~0.005), so it's a faithful stand-in reference for measuring how much the smaller quants drift. KLD values here are therefore **relative to Q80** (Q8_0 = 0 by definition).
ModelSize GBKLD meanKLD p95KLD maxTop-1 match
Q8_0 (reference)40.00.00000.00000.0000100.0%
Q5KM26.780.00980.04084.89396.61%
Q4KM22.900.03370.15054.10593.94%
Q3KM18.670.11140.54197.90488.89%
IQ2_M13.540.37991.917812.25977.82%

<sub>Full per-quant reports in `reports/`; machine-readable summary in `metrics/quant-summary.csv`; SHA-256 of every file in `reports/artifact-sha256sums.txt`.</sub>

📈 Charts

[image] [image]

🧮 Will it fit? (RAM cheat-sheet)

A 46B is memory-hungry; add KV-cache on top (it grows with context — 160K is a lot). | you have | quant | context | |---|---|---| | 24 GB | IQ2M | ~8–16K | | 32 GB | Q3KM / IQ2M | ~16–32K | | 48 GB | Q4KM / Q5KM | ~32–64K | | 64 GB+ | Q8_0 | large (up to 160K with room) |

🚀 How to run it

bash
# ollama
ollama run hf.co/KikoCis/ALIA-40b-instruct-2601-GGUF:Q4_K_M

# llama.cpp
llama-server -m ALIA-40b-instruct-2601-Q4_K_M.gguf -c 32768 --jinja -ngl 99

Recommended sampling: temperature ~0.7, top_p ~0.9. Chat/instruct model (uses its built-in template) — great for Spanish and the co-official languages (Catalan, Galician, Basque) + European languages.

⚠️ Good to know

  • —Strengths: a genuinely sovereign, EU-built 46B — strong Spanish + co-official + European multilinguality, 160K context, permissive licence.
  • —Limits: it's a 46B — even the smallest quant needs ~24 GB RAM; IQ2_M trades real quality for size. Not a specialised coding model.
  • —KLD is measured vs Q8_0 (F16 doesn't fit this GPU) — see the note above.

📊 Evaluation methodology

  • —What: quantization fidelity vs the Q8_0 reference — llama-perplexity --kl-divergence (KLD mean/p95/max, ΔPPL, top-1 agreement).
  • —Corpus: Spanish text, ctx 512, same corpus used for imatrix calibration.
  • —Reference: Q8_0 (near-lossless stand-in for F16, which is too large for this GPU).
  • —Date: 2026-07. Caveat: relative fidelity ranking across quants of this model.

🔁 Provenance & reproducibility

  • —Scripts: `scripts/` — exact convert → Q8 → imatrix → quant → KLD commands.
  • —imatrix: `alia-40b-es.imatrix` — importance matrix (Spanish calibration).
  • —Checksums: `reports/artifact-sha256sums.txt`.
  • —Source: `BSC-LT/ALIA-40b-instruct-2601` — weights unmodified (faithful quantization).

🗒️ Changelog

  • —2026-07 v1: initial imatrix GGUF ladder (IQ2M → Q80) + KLD metrics, Spanish-calibrated.

📚 Credit & license

Model, weights, training: © Barcelona Supercomputing Center — the ALIA / langtech-bsc project (model · ALIA-kit). Quant ladder + imatrix (es) + KLD metrics: KikoCis. Apache-2.0 (same as upstream). No weights modified.