FrontiersMind/Lumma-0.6B-Base

<p align="center">

Lumma-0.6B-Base

A multilingual language model optimized for efficient deployment and English–Indic language understanding.

600M Parameters • 1 Trillion Training Tokens • 12,288 Context Length • Shared KV

</p>

Supported Languages

The model is trained on English and a diverse set of Indic languages, including:

English, Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia

Overview

Lumma-0.6B-Base is a multilingual decoder-only language model trained from scratch on 1 trillion tokens. It is designed for efficient deployment, long-context inference, and strong multilingual performance across English and Indic languages, featuring a compact transformer architecture, memory-efficient attention mechanisms, and an optimized multilingual tokenizer.

Key Features

• —Trained from scratch on 1 trillion tokens
• —600 million parameter decoder-only Transformer
• —Native English and Indic language pretraining
• —Shared KV Attention for memory-efficient inference
• —12,288 token context length
• —Grouped Query Attention (GQA)
• —RMSNorm with QK Normalization
• —SwiGLU feed-forward network
• —Factorized tied embeddings
• —Large multilingual tokenizer optimized for Indic languages

[!NOTE] We do not recommend using base language models for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.

Shared KV

Lumma introduces Shared KV, an alternative key-value caching strategy designed to reduce inference memory requirements without significantly impacting model quality.

Instead of computing independent Key and Value projections, both are derived from a shared latent representation. During attention computation, lightweight Key normalization and RoPE transformations are applied dynamically.

This approach reduces KV-cache memory usage by approximately 50%, making Lumma better suited for long-context inference and memory-constrained deployments.

<p align="center"> <img src="./sharedkvcachecomparisonimproved.png" width="650"/> </p>

KV Cache Modes

Lumma supports two inference modes depending on deployment requirements.

Shared KV

model.config.kv_cache_mode = "shared"

Recommended when memory is the primary bottleneck.

• —Approximately 50% lower KV-cache memory
• —Slightly higher compute overhead
• —Better suited for long-context inference

Vanilla KV

model.config.kv_cache_mode = "vanilla"

Recommended for standard deployments.

• —Standard KV-cache implementation
• —Lower compute overhead
• —Maximum compatibility across inference frameworks

Benchmark Results

The following results correspond to the released Lumma-0.6B model trained on 1 trillion tokens.

General Benchmarks

<div align="center">

<table> <thead> <tr> <th>Model Name</th> <th>Tokens Budget<br>(Trillion)</th> <th>HellaSwag</th> <th>Winogrande</th> <th>OBQA</th> <th>ARC-e</th> <th>ARC-c</th> <th>Average</th> </tr> </thead> <tbody> <tr> <td>MobiLlama-0.5B-Base</td> <td align="center">1.3</td> <td align="center">39.65</td> <td align="center">53.67</td> <td align="center">30.60</td> <td align="center">52.82</td> <td align="center">23.63</td> <td align="center">40.07</td> </tr> <tr> <td>Qwen-2-0.5-Base</td> <td align="center">12</td> <td align="center">49.01</td> <td align="center">57.69</td> <td align="center">33.20</td> <td align="center">54.79</td> <td align="center">25.42</td> <td align="center">44.02</td> </tr> <tr> <td>Qwen2.5-0.5B-Base</td> <td align="center">18</td> <td align="center">52.16</td> <td align="center">56.82</td> <td align="center">35.40</td> <td align="center">64.64</td> <td align="center">29.86</td> <td align="center">47.78</td> </tr> <tr> <td><strong>Lumma-0.6B-Base</strong></td> <td align="center"><strong>1</strong></td> <td align="center">46.25</td> <td align="center">54.14</td> <td align="center">32.80</td> <td align="center">60.60</td> <td align="center">28.58</td> <td align="center">44.47</td> </tr> </tbody> </table>

</div>

Multilingual Tokenization

Efficient tokenization is particularly important for multilingual language models.

Lower fertility indicates fewer tokens are required to represent text, improving both training efficiency and inference cost.

Usage

!pip install transformers=='5.4.0'


from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "FrontiersMind/Lumma-0.6B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.bfloat16
).eval()

# Memory-efficient mode
model.config.kv_cache_mode = "shared"

# Standard mode
# model.config.kv_cache_mode = "vanilla"

prompt = "The world is a strange place"

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.3,
    top_p=0.95,
    top_k=20,
    repetition_penalty=1.1,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@misc{lumma2026,
  title={Lumma-0.6B},
  author={FrontiersMind},
  year={2026},
  url={https://huggingface.co/FrontiersMind/Lumma-0.6B}
}