montevive/ALIA-40b-instruct-2601-NVFP4-GGUF

ALIA-40b-instruct-2601 — NVFP4 GGUF

NVFP4 quantization of BSC-LT/ALIA-40b-instruct-2601 packaged for llama.cpp. 27 GB on disk, ~10 tok/s generation on a single NVIDIA GB10 (DGX Spark), with full multilingual capability across the Iberian languages the base model was trained on.

For the vLLM / TensorRT-LLM (compressed-tensors safetensors) version of this model, see montevive/ALIA-40b-instruct-2601-NVFP4.

What's in this repo

• —The GGUF itself — produced by NVIDIA ModelOpt (NVFP4 calibration on 128 samples of `HuggingFaceH4/ultrachat_200k` at sequence length 2048), then re-packed into GGUF with a patched convert_hf_to_gguf.py (see below).
• —`convert_hf_to_gguf.NVFP4-rope-permute-fix.patch` — the 14-line convert_hf_to_gguf.py patch needed to produce a correct NVFP4 GGUF for any Llama-architecture model. Adds a _repack_nvfp4 override on LlamaModel that mirrors the BF16 permutation site in modify_tensors. Without it the conversion silently produces gibberish output. Submitted upstream as ggml-org/llama.cpp#22611.
• —`docs/conversion-and-fix.md` — full debugging notes: how the bug was found, why it only affects NVFP4 (other quants like Q4KM / Q5KM are unaffected), and how to reproduce on TinyLlama-1.1B in ~10 seconds.

Usage

Requires a recent llama.cpp build (NVFP4 type was added in PR #19769, March 2026). On NVIDIA Blackwell hardware (compute capability ≥ 12.0), PR #22196 provides native tensor-core acceleration.

# build llama.cpp with CUDA
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

# run on GPU — use --jinja so llama.cpp picks up ALIA's embedded chat template.
# (Do NOT use --chat-template chatml: ALIA's training format folds the system
#  prompt into the first user turn rather than emitting a standalone <|im_start|>system
#  block, and default ChatML silently degrades quality. See PR #1 in the source repo.)
./build/bin/llama-cli \
    -m ALIA-40b-instruct-2601.NVFP4.gguf \
    -ngl 99 -c 4096 \
    --jinja -cnv

# or batch completion
./build/bin/llama-completion \
    -m ALIA-40b-instruct-2601.NVFP4.gguf \
    -ngl 99 -c 1024 -n 200 --temp 0 \
    -p "La capital de España es"

Recommended sampling (per the base model card): temperature between 0 and 0.2; avoid repetition penalties — they degrade instruction-following on this model.

Performance

Measured on NVIDIA DGX Spark (GB10, 128 GB unified memory, aarch64), 113-token Spanish prompt, greedy decoding, 200 tokens generated, llama.cpp commit aab68217:

On Blackwell GPUs NVFP4 is the right choice — ~1.9× faster generation than Q80, **33% smaller** on disk and in VRAM. On non-Blackwell hardware (or CPU-only deployment), Q80 / Q5KM / Q4KM from the BF16 source remain better trade-offs because their kernels are far more mature.

Quality

NVFP4 incurs more quantization noise than Q8_0 (which is essentially lossless), but ALIA-40B is well within NVFP4's design sweet spot (originally targeted at 70B+ models). Multilingual smoke tests on the converted GGUF:

> La capital de España es 
La capital de España es Madrid. Es la ciudad más grande y poblada del país,
y se encuentra en el centro de la Península Ibérica. Madrid es conocida por
su rica historia, su cultura vibrante y su vida nocturna animada...

> Quina és la capital de Catalunya?
La capital de Catalunya és Barcelona. És una ciutat vibrant, cosmopolita i 
plena de cultura, amb una població de més de 1,6 milions d'habitants...

> Calcula: 125 × 8 + 50 = 
125 × 8 + 50 = (100 × 8 + 25 × 8) + 50 = ...

Calibration caveat

Calibration used HuggingFaceH4/ultrachat_200k — an English-only synthetic conversation dataset. ALIA is BSC's Iberian-multilingual model, so this is suboptimal: the per-tensor and per-block scales were computed against an activation distribution that doesn't fully represent the model's actual use case. Re-quantizing with a multilingual instruction dataset (Iberian languages + code) would likely improve quality on Spanish/Catalan/Basque/Galician outputs. We did not have time to do this for this release; PRs welcome.

Important: the conversion fix

convert_hf_to_gguf.py had a bug specific to the NVFP4 path: Llama-architecture q_proj and k_proj weights need an axis-0 row permutation to match GGML's RoPE convention. The BF16 path applies this in LlamaModel.modify_tensors, but the NVFP4 path bypasses modify_tensors and writes weights directly via _repack_nvfp4. Without the permutation, attention heads get scrambled at inference and the model produces gibberish (e.g., Certainlyrics|assistant|assistant|... for "The capital of France is").

The convert_hf_to_gguf.NVFP4-rope-permute-fix.patch file in this repo applies the missing permutation to both the nibble-packed weight and the per-block scale tensor. With the fix, perplexity on a TinyLlama-1.1B reproducer drops from PPL 4419 (gibberish) to PPL 43.9 (matches the BF16-dequantized baseline of 44.0).

Submitted upstream: ggml-org/llama.cpp#22611. Once merged, the patch in this repo will be obsolete for any llama.cpp build that includes it.

If you want to reproduce or re-quantize from scratch:

# 1. Clone llama.cpp at the same commit (or any later commit that doesn't fix this upstream)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout aab68217   # the commit this GGUF was built against

# 2. Apply the fix
patch -p1 < /path/to/convert_hf_to_gguf.NVFP4-rope-permute-fix.patch

# 3. Quantize the BF16 source with NVIDIA ModelOpt (separate venv recommended)
#    See https://github.com/NVIDIA/TensorRT-Model-Optimizer for ModelOpt setup.
#    Reference scripts: https://github.com/montevive/alia-nvfp4-research

# 4. Convert ModelOpt safetensors to GGUF
python convert_hf_to_gguf.py /path/to/nvfp4-modelopt/ \
    --outfile ALIA-40b-instruct-2601.NVFP4.gguf

License & attribution

This quantized model is released under the same Apache 2.0 license as the source.

• —Base model: BSC-LT/ALIA-40b-instruct-2601 by Barcelona Supercomputing Center (BSC). Please cite their work if you use this model in research:

  @misc{alia-40b-instruct,
    author = {Barcelona Supercomputing Center},
    title  = {ALIA-40b-instruct},
    year   = {2026},
    url    = {https://huggingface.co/BSC-LT/ALIA-40b-instruct-2601}
  }

• —NVFP4 conversion + bug fix: Montevive AI. The convert_hf_to_gguf.py patch is offered for upstream contribution to ggml-org/llama.cpp.

Limitations

Inherits all limitations of the base ALIA-40b-instruct-2601 model. Additionally:

• —NVFP4 inference quality is below Q8_0 on small models. Empirically fine on 40B but you should evaluate on your own task before deploying.
• —NVFP4 GGUF support in llama.cpp is recent (March 2026, PR #19769). Expect API churn and potential further fixes upstream. The fix patch in this repo may become unnecessary if it is merged upstream.
• —CPU performance is poor compared to other quants — the ARM NEON dot product for NVFP4 is much newer than for Q8_0 / K-quants. Use Blackwell GPUs for real workloads.
• —The model was calibrated on English data; multilingual tasks may benefit from re-quantization with Iberian-language calibration data.