trillionlabs/Trida-7B

1---2license: apache-2.03tags:4- finetuned5- chat6language:7- en8- ko9- ja10pipeline_tag: text-generation11library_name: transformers12extra_gated_fields:13  Full Name: text14  Email: text15  Organization: text16---17 18# Trida-7B19 20## Introduction21 22🚀 Trida-7B: Block Diffusion Language Model23 24We introduce Trida-7B, a high-performance 7-billion parameter language model representing the first publicly released Block Diffusion Language Model to originate from Korea.25 26### Model Overview27Architecture: Block Diffusion Language Model28 29Base Model: Continually pre-trained from Tri-7B model.30 31 32Korean Language Leadership33Trida-7B sets a new benchmark for generative models in the region. To our knowledge, it is the:34 35- First Block Diffusion Language Model to be openly released in Korea.36 37- First Block Diffusion Language Model trained with Step-wise autoregressive attention.38 39- Best-performing diffusion language model in Korean among similar model sizes.40 41This model is a significant step forward for the Korean LLM community, demonstrating the effectiveness of the Block Diffusion paradigm for complex, multilingual tasks.42 43 44 45### Key Highlights46* **Block Diffusion Architecture**: Trida-7B leverages the **Block Diffusion** architecture, combining the strengths of **parallelized diffusion generation** with **autoregressive dependencies** for improved efficiency, control, and flexible-length sequence generation.47* **Step-wise Autoregressive Attention** An attention mechanism that enables single-pass training and efficient RL by fixing attention masks during the unmasking process. Also improves inference efficiency by enabling kv-caching within the current block.48* **Multilingual Leadership**: Specially optimized for **Korean, English, and Japanese**, offering robust performance across all three languages.49* **Korean First**: To our knowledge, Trida-7B-Preview is the **first Block Diffusion Language Model** to be openly released in Korea.50* **Best-in-Class Korean Performance**: It is the **best-performing diffusion language model in Korean** among models of similar size, setting a new benchmark for generative models in the region.51 52 53### Model Specifications54 55#### Trida-7B56- Type: Block Diffusion Language Model57- Training Stage: Pre-training & Post-training58- Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm59- Number of Parameters: 7.76B60- Number of Layers: 3261- Number of Attention Heads: 3262- Context Length: 8,19263- Vocab Size: 128,25664 65#### 🔄 Training and Methodology66 67Continual Pre-training from Tri-7B:68Rather than training from scratch, Trida-7B was developed through Continual Pre-training from our state-of-the-art autoregressive model, trillionlabs/Tri-7B.69- Knowledge Transfer: To prevent catastrophic forgetting during the transition from AR to Diffusion, we employed blocksize warmup.70 71Step-wise Autoregressive Attention for Efficient RL & Inference72One of the most significant innovations in Trida-7B is the Step-wise Autoregressive Attention mechanism. This design solves the primary bottleneck of Diffusion models: the need for $T$ sequential forward passes during generation and Reinforcement Learning (RL).73- Mechanism: During the rollout process, we fix the attention mask for each token at the exact moment it is "unmasked." This creates a structured, causal-like dependency within a single sequence.74- Single-pass Training: By aligning the denoising steps into a step-wise autoregressive structure, we enable the model to calculate gradients for all denoising steps in a single forward/backward pass.75- Impact: This reduces the computational overhead of RL and iterative inference by up to $1/T$, allowing Trida-7B to achieve training and inference speeds much faster than traditional Autoregressive models while maintaining the diverse generative capabilities of Diffusion.76 77 78## 🚀 Quickstart79 80 81 82```python83from transformers import AutoModelForCausalLM, AutoTokenizer84 85model_name = "trillionlabs/Trida-7B"86 87model = AutoModelForCausalLM.from_pretrained(88    model_name,89    torch_dtype="auto",90    device_map="auto",91    trust_remote_code=True92)93 94tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)95 96prompt = "Explain the Korean concept of 'Sonnim' (guest) and compare it to Japanese 'Omotenashi' in English."97messages = [98    {"role": "system", "content": "You are Trida, created by TrillionLabs. You are a helpful assistant."},99    {"role": "user", "content": prompt}100]101 102text = tokenizer.apply_chat_template(103    messages,104    tokenize=False,105    add_generation_prompt=True106)107inputs = tokenizer([text], return_tensors="pt").to(model.device)108 109# Block Step-wise Autoregressive Generation110gen_ids = model.generate(111    inputs["input_ids"],112    tokenizer=tokenizer,113    max_new_tokens=4096,114    threshold=0.9,115)116 117response = tokenizer.decode(118    gen_ids[0][inputs["input_ids"].shape[1]:], 119    skip_special_tokens=True120)121print(response)122```123 124You can also checkout our repo (https://github.com/trillion-labs/Fast-dLLM-Trida) for evaluation and demo.125 126 127Our full technical blog post is coming soon—stay tuned!128 129---130 131 132 133## Evaluation134 135We evaluated Trida-7B across a comprehensive suite of benchmarks assessing general reasoning, knowledge recall, coding abilities, mathematical reasoning, and instruction-following capabilities.136 137<details>138<summary> Full evaluation settings </summary>139 140| Benchmark | Language | Evaluation Setting | Metric |141|:----------|:---------|:------------------|:-------|142| **General Reasoning and Factuality** | | | |143| • xwinograd_en | English | 0-shot | accuracy |144| • xwinograd_jp | Japanese | 0-shot | accuracy |145| • KoBEST | Korean | 5-shot | accuracy |146| **Knowledge and Reasoning** | | | |147| • KMMLU | Korean | 5-shot | accuracy |148| • MMLU | English | 5-shot | accuracy |149| • Global-MMLU-Lite-en | English | 5-shot | accuracy |150| • Global-MMLU-Lite-ko | English | 5-shot | accuracy |151| • Global-MMLU-Lite-ja | Japanese | 5-shot | accuracy |152| • BBH | English | 3-shot, CoT | accuracy |153| • MMLU pro | English | 0-shot, CoT | accuracy |154| **Coding** | | | |155| • HumanEval | English | 0-shot | pass@1 |156| • MBPPPlus | English | 0-shot | pass@1 |157| • KoMBPPPlus | Korean | 0-shot | pass@1 |158| **Mathematical Reasoning** | | | |159| • GSM8k | English | 0-shot, CoT | exact-match |160| • KoGSM8k | Korean | 0-shot, CoT | exact-match |161| • MATH500 | English | 0-shot, CoT | exact-match |162| **Instruction Following and Chat** | | | |163| • IFEval | English | 0-shot | strict-prompt |164| • koIFEval | Korean | 0-shot | strict-prompt |165 166 167</details>168 169### Benchmark Results170 171### General Reasoning and Factuality172| Benchmark | Trida-7B |173| --- | --- |174| KoBEST | 74.08 |175| KMMLU | 50.28 |176| MMLU | 67.23 |177| Global-MMLU-Lite-en | 73.5 |178| Global-MMLU-Lite-ko | 64.25 |179| Global-MMLU-Lite-ja | 64.25 |180| xwinograd_en | 69.81 |181| xwinograd_jp | 64.75 |182| BBH | 52.45 |183| MMLU pro | 39.37 |184 185### Coding186| Benchmark | Trida-7B |187| --- | --- |188| HumanEval | 35.98 |189| MBPP Plus | 50.79 |190| KoMBPP Plus | 46.3 |191 192### Mathematical Reasoning193| Benchmark | Trida-7B |194| --- | --- |195| GSM8k | 65.13 |196| KoGSM8k | 61.26 |197| MATH500 | 33.6 |198 199### Instruction Following200| Benchmark | Trida-7B |201| --- | --- |202| IFEval | 64.98 |203| koIFEval | 61.74 |204 205 206### Korean Performance Vs Other Diffusion LLMs207| Benchmark | Trida-7B | Llada-7B | Dream-7B | Fast-dllm-v2 |208| --- | --- | --- | --- | --- | 209| KoMBPP Plus (pass@1) | 46.3 | 5.8 | 56.61 | 67.2 |210| koIFEval (prompt-strict) | 53.42 | 22.4| 8.9| 46.17 |  211| koGSM8K (strict extract accuracy) | 61.26 | 38.6 | 25.02 | 56.94 | 212| kobest (accuracy) | 74.92 | 54.55 | 61.92 | 57.22 | 213| KMMLU (accuracy) | 46.35 | 29.33 | 39.84 | 44.36 |214| Global-MMLU-Lite-ko (accuracy)| 60.25 | 20.12 | 55.25 | 55.0 |215| avg| 57.08 |28.47 | 41.26 | 54.48  |216 217 218 219 220## Limitations221 222- Language Support: The model is optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.223- Knowledge Cutoff: The model's information is limited to data available up to Febuary, 2025.224 225## License226This model is licensed under the Apache License 2.0.227 228 229## Contact230For inquiries, please contact: info@trillionlabs.co