CollectionStudio/Trinity-Nano-Base-Pre-Anneal

1---2license: other3language:4- en5- es6- fr7- de8- it9- pt10- ru11- ar12- hi13- ko14- zh15library_name: transformers16license_link: LICENSE17license_name: openmdw-1.118---19<div align="center">20  <picture>21    <img22      src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/i-v1KyAMOW_mgVGeic9WJ.png"23      alt="Arcee Trinity Nano"24      style="max-width: 100%; height: auto;"25    >26  </picture>27</div>28 29# Trinity Nano Base Pre Anneal30 31Trinity-Nano-Base-Pre-Anneal is an Arcee AI 6B MoE model with 1B active parameters. It is the small-sized model in our new Trinity family, a series of open-weight models for enterprise and tinkerers alike.32 33This base model is a pre-anneal checkpoint captured at Adam LR: 0.0002, Muon LR: 0.001 before starting learning rate decay on a high-quality data mix.34While this checkpoint was not exposed to the anneal phase mix containing high proportions of math and code content, it has been trained on significant amounts of such data.35This checkpoint is not suitable for chatting or general use without further finetuning and should be trained for your specific domain before use.36 37***38 39Trinity-Nano-Base-Pre-Anneal is trained on 8.8T tokens gathered and curated through a key partnership with [Datology](https://www.datologyai.com/), building upon the excellent dataset we used on [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) with additional math and code.40 41Training was performed on a cluster of 512 H200 GPUs powered by [Prime Intellect](https://www.primeintellect.ai/) using HSDP parallelism.42 43More details, including key architecture decisions, can be found on our blog [here](https://www.arcee.ai/blog/the-trinity-manifesto)44 45***46 47## Model Details48 49* **Model Architecture:** AfmoeForCausalLM50* **Parameters:** 6B, 1B active51* **Experts:** 128 total, 8 active, 1 shared52* **Context length:** 4K53* **Learning rate during pretraining**:54  * `adam_lr = 0.0002`55  * `muon_lr = 0.001`56* **Training Tokens:** 8.8T57* **License:** [OpenMDW-1.1](https://huggingface.co/arcee-ai/Trinity-Mini#license)58 59***60 61<div align="center">62  <picture>63      <img src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/sSVjGNHfrJKmQ6w8I18ek.png" style="background-color:ghostwhite;padding:5px;" width="17%" alt="Powered by Datology">64  </picture>65</div>66 67## Try out our reasoning tune of our medium-sized Trinity Mini model68 69Trinity Mini is available today on openrouter:70 71https://openrouter.ai/arcee-ai/trinity-mini72 73```74curl -X POST "https://openrouter.ai/v1/chat/completions" \75  -H "Authorization: Bearer $OPENROUTER_API_KEY" \76  -H "Content-Type: application/json" \77  -d '{78    "model": "arcee-ai/trinity-mini",79    "messages": [80      {81        "role": "user",82        "content": "What are some fun things to do in New York?"83      }84    ]85  }'86```87 88 89## License90 91Trinity-Nano-Base-Pre-Anneal is released under the OpenMDW-1.1 license.