arcee-ai/Trinity-Mini-NVFP4

1---2license: other3language:4- en5- es6- fr7- de8- it9- pt10- ru11- ar12- hi13- ko14- zh15library_name: transformers16base_model:17- arcee-ai/Trinity-Mini18base_model_relation: quantized19tags:20- moe21- nvfp422- modelopt23- blackwell24- vllm25license_link: LICENSE26license_name: openmdw-1.127---28<div align="center">29  <picture>30    <img31      src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/i-v1KyAMOW_mgVGeic9WJ.png"32      alt="Arcee Trinity Mini"33      style="max-width: 100%; height: auto;"34    >35  </picture>36</div>37 38# Trinity Mini NVFP439 40**This repository contains the NVFP4 quantized weights of Trinity-Mini for deployment on NVIDIA Blackwell GPUs.**41 42Trinity Mini is an Arcee AI 26B MoE model with 3B active parameters. It is the medium-sized model in our new Trinity family, a series of open-weight models for enterprise and tinkerers alike.43 44This model is tuned for reasoning, but in testing, it uses a similar total token count to competitive instruction-tuned models.45 46***47 48Trinity Mini is trained on 10T tokens gathered and curated through a key partnership with [Datology](https://www.datologyai.com/), building upon the excellent dataset we used on [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) with additional math and code.49 50Training was performed on a cluster of 512 H200 GPUs powered by [Prime Intellect](https://www.primeintellect.ai/) using HSDP parallelism.51 52More details, including key architecture decisions, can be found on our blog [here](https://www.arcee.ai/blog/the-trinity-manifesto)53 54***55 56## Model Details57 58* **Model Architecture:** AfmoeForCausalLM59* **Parameters:** 26B, 3B active60* **Experts:** 128 total, 8 active, 1 shared61* **Context length:** 128k62* **Training Tokens:** 10T63* **License:** [OpenMDW-1.1](https://huggingface.co/arcee-ai/Trinity-Mini#license)64* **Recommended settings:**65    * temperature: 0.1566    * top_k: 5067    * top_p: 0.7568    * min_p: 0.0669 70***71 72## Benchmarks73 74![](https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/UMV0OZh_H1JfvgzBTXh6u.png)75 76<div align="center">77  <picture>78      <img src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/sSVjGNHfrJKmQ6w8I18ek.png" style="background-color:ghostwhite;padding:5px;" width="17%" alt="Powered by Datology">79  </picture>80</div>81 82## Quantization Details83 84- **Scheme:** NVFP4 (`nvfp4_mlp_only` — MLP/expert weights only, attention remains BF16)85- **Tool:** [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)86- **Calibration:** 512 samples, seq_length=2048, all-expert calibration enabled87- **KV cache:** Not quantized88 89## Running with vLLM90 91Requires [vLLM](https://github.com/vllm-project/vllm) >= 0.18.0. Native FP4 compute requires Blackwell GPUs; older GPUs fall back to Marlin weight decompression automatically.92 93### Blackwell GPUs (B200/B300/GB300) — Docker (recommended)94 95```bash96docker run --runtime nvidia --gpus all -p 8000:8000 \97  -v ~/.cache/huggingface:/root/.cache/huggingface \98  vllm/vllm-openai:v0.18.0-cu130 \99  arcee-ai/Trinity-Mini-NVFP4 \100  --trust-remote-code \101  --gpu-memory-utilization 0.90 \102  --max-model-len 8192103```104 105### Hopper GPUs (H100/H200) and others106 107```bash108vllm serve arcee-ai/Trinity-Mini-NVFP4 \109  --trust-remote-code \110  --gpu-memory-utilization 0.90 \111  --max-model-len 8192 \112  --host 0.0.0.0 \113  --port 8000114```115 116 **Note (Blackwell pip installs):** If installing vLLM via pip on Blackwell rather than using Docker, native FP4 kernels may produce incorrect output due to package version mismatches. As a workaround, force the Marlin backend:117 118 ```bash119 export VLLM_NVFP4_GEMM_BACKEND=marlin120 121 vllm serve arcee-ai/Trinity-Mini-NVFP4 \122   --trust-remote-code \123   --moe-backend marlin \124   --gpu-memory-utilization 0.90 \125   --max-model-len 8192 \126   --host 0.0.0.0 \127   --port 8000128 ```129 130Marlin decompresses FP4 weights to BF16 for compute, providing the full memory compression benefit (~3.7× vs BF16) but not native FP4 compute speedup. On Hopper GPUs (H100/H200), Marlin is selected automatically and no extra flags are needed.131 132## License133 134Trinity-Mini-NVFP4 is released under the OpenMDW-1.1 license.