Modelpublic

arcee-ai/Trinity-Large-Base

sourceHugging Faceotherupdated 4mo agoView on Hugging Face

57likes299downloads

Model Card

<div align="center"> <picture> <img src="https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/i-v1KyAMOW_mgVGeic9WJ.png" alt="Arcee Trinity Large" style="max-width: 100%; height: auto;"

</picture> </div> <hr>

Trinity-Large-Base

Introduction

Trinity-Large-Base is a pretrained foundation model from Arcee AI's Trinity Large training run. It is a 398B-parameter sparse Mixture-of-Experts (MoE) model with approximately 13B active parameters per token. The checkpoint was captured after 17 trillion tokens of pretraining, including mid-training learning-rate anneals and context extension, but prior to any instruction tuning or reinforcement learning.

This checkpoint represents the completed pretraining phase and serves as a foundation for research and downstream fine-tuning.

More details on the training of Trinity Large are available in the technical report.

Model Variants

The Trinity Large family consists of three checkpoints from the same training run:

—Trinity-Large-Base (this release): Full 17T-token pretrained foundation model with mid-training anneals
—[Trinity-Large-Thinking](https://huggingface.co/arcee-ai/Trinity-Large-Thinking): Reasoning-optimized, agentic post-training with extended chain-of-thought
—[Trinity-Large-TrueBase](https://huggingface.co/arcee-ai/Trinity-Large-TrueBase): 10T-token pre-anneal checkpoint with no instruction data
—[Trinity-Large-Preview](https://huggingface.co/arcee-ai/Trinity-Large-Preview): Lightly post-trained, chat-ready model undergoing active RL

Architecture

Trinity-Large-Base uses a sparse MoE configuration designed to maximize efficiency while maintaining large-scale capacity.

Hyperparameter	Value
Total parameters	~398B
Active parameters per token	~13B
Experts	256
Active experts	4
Routing strategy	4-of-256 (1.56% sparsity)
Dense layers	6
Pretraining context length	8,192
Context length after extention	512k
Architecture	Sparse MoE (AfmoeForCausalLM)

Benchmark Results

Benchmark	N-shot	Metric	Score	Stderr
mbpp_plus	3	passat1,none	0.8862	±0.0164
minerva_math500	4	math_verify,none	0.6520	±0.0213
hellaswag_5shot	5	acc_norm,none	0.9011	±0.0030
winogrande_5shot	5	acc,none	0.8082	±0.0111
mmlu_5shot	5	acc,none	0.8258	±0.0031
mmlugenerative5shot	5	exactmatch,getresponse	0.8260	±0.0031
mmlu_pro	5	exact_match,custom-extract	0.6602	±0.0042
triviaqa_5shot	5	exactmatch,removewhitespace	0.8330	±0.0028
arcchallenge0shot	0	acc_norm,none	0.6544	±0.0139
bbh_fewshot	3	exactmatch,removewhitespace	0.6570	±0.0051
gpqadiamond5shot	5	acc_norm,none	0.4394	±0.0354
gsm8k_cot	8	exact_match,flexible-extract	0.9136	±0.0077

Training Configuration

Pretraining

—Training tokens: 17 trillion
—Checkpoint type: Post-anneal (foundation)
—Instruction data: None
—RLHF or post-training: None

This checkpoint represents the final pretrained state after completion of the pretraining phase, including mid-training learning-rate anneals, but before instruction tuning or reinforcement learning.

Optimizers

Optimizer learning rates during WSD stable phase:

—Adam learning rate: 2e-4
—Muon learning rate: 8e-4

Muon was used to support larger critical batch sizes in a highly sparse MoE regime.

Infrastructure

—Hardware: 2,048 NVIDIA B300 GPUs
—Parallelism: HSDP + Expert Parallelism
—Compute partner: Prime Intellect
—Data partner: Datology

Intended Use

—Studying emergent behavior from large-scale pretraining
—Sparse MoE routing and load-balancing research
—Interpretability, probing, and ablation studies
—Domain-specific fine-tuning from a pretrained foundation
—Academic and industrial foundation model research

Comparison with TrueBase

Trinity-Large-Base includes an additional 7 trillion training tokens compared to Trinity-Large-TrueBase, along with mid-training learning-rate anneals. These anneals stabilize training dynamics and typically improve downstream fine-tuning performance compared to the pre-anneal checkpoint. Researchers studying raw pretraining dynamics may prefer TrueBase, while those seeking a foundation for fine-tuning may prefer this checkpoint.

Known Limitations

—Not aligned for safety, helpfulness, or conversational tone
—Requires substantial compute and expertise to fine-tune
—May exhibit raw or unstable behaviors typical of unaligned models
—No extended-context tuning beyond the 8K pretraining window

License

Trinity-Large-Base is released under the OpenMDW License, version 1.1 (OpenMDW-1.1).