janihal/Llama-Poro-2-8B-Instruct-oQ4e

Llama-Poro-2-8B-Instruct-oQ4e

A 4-bit (mixed-precision) MLX quantization of **LumiOpen/Llama-Poro-2-8B-Instruct**, produced with oQ (oMLX v0.6.4) importance-matrix quantization. It is intended for fast local inference on Apple Silicon.

An 8-bit sibling is available at janihal/Llama-Poro-2-8B-Instruct-oQ8e.

Original model

Poro 2 8B Instruct is a Finnish/English instruction-following model built on the Llama 3.1 8B architecture through continued pretraining, SFT and DPO. It was created by AMD Silo AI, the TurkuNLP group at the University of Turku, and HPLT, and trained on the LUMI supercomputer.

All model behaviour, capabilities, evaluation results, training data, intended use and limitations are documented in the [original model card](https://huggingface.co/LumiOpen/Llama-Poro-2-8B-Instruct). This repository only changes the weight precision and storage format — please refer to the original for everything else.

Quantization details

• —Base model: LumiOpen/Llama-Poro-2-8B-Instruct
• —Tool: oQ / oMLX v0.6.4
• —Architecture: llama (8.03B params, 32 layers, 128256 vocab, 8192 context)
• —Precision: 4-bit affine, mixed-precision — most weights at 4-bit, with attention projections and early-layer mlp.down_proj kept at 5–6 bit for quality
• —Group size: 64
• —Calibration: importance matrix (imatrix) enabled, oqe_code_multilingual calibration set, 128 samples at sequence length 512
• —Format: MLX safetensors
• —Size on disk: ~4.4 GB

Per-tensor bit assignments are recorded in config.json; calibration metadata is in oq_imatrix_report.json.

Usage

Requires `mlx-lm` on an Apple Silicon Mac.

pip install mlx-lm

mlx_lm.generate --model janihal/Llama-Poro-2-8B-Instruct-oQ4e \
  --prompt "Mikä on Suomen pääkaupunki?"

from mlx_lm import load, generate

model, tokenizer = load("janihal/Llama-Poro-2-8B-Instruct-oQ4e")
messages = [{"role": "user", "content": "Kerro lyhyesti poroista."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

The chat template is bundled in tokenizer_config.json.

License

Released under the Llama 3.3 Community License, inherited from the base model. Built with Llama.

Attribution

If you use this model, please cite the original Poro 2 work by LumiOpen / AMD Silo AI / TurkuNLP / HPLT.