JetBrains/Mellum2-12B-A2.5B-Base

<img alt="Mellum" src="mellum-logo-dark.svg" width="320">

Mellum2 Base

[!Note] Use this checkpoint as the starting point for your own fine-tuning, alignment, or domain adaptation on top of the long-context base. For instruction-following or reasoning tasks out of the box, use Instruct or Thinking instead.

Mellum2 Base Highlights

Mellum2 Base is a long-context pretrained causal language model trained by JetBrains.

The model uses a Mixture-of-Experts architecture with 64 experts and activates 8 experts per token. It uses a combination of sliding-window and full attention layers, with a context length of 131,072 tokens.

This is the long-context base, produced from `Mellum2-12B-A2.5B-Base-Pretrain` by a layer-selective YaRN extension stage that re-maps RoPE frequencies on the global-attention layers only. It is the shared starting point for the released Instruct and Thinking variants.

Mellum2 Model Family

This repository contains one checkpoint from the Mellum2 family.

Model Overview

Mellum2 Base has the following features:

• —Number of Layers: 28
• —Hidden Size: 2304
• —Intermediate Size: 7168
• —MoE Intermediate Size: 896
• —Number of Experts: 64
• —Number of Activated Experts: 8
• —Number of Attention Heads (GQA): 32 for Q and 4 for KV
• —Context Length: 131,072
• —Sliding Window: 1,024
• —Vocabulary Size: 98,304
• —Precision: bfloat16

Serving with vLLM

vllm serve JetBrains/Mellum2-12B-A2.5B-Base --max-model-len 131072

Quickstart

Text-Only Input (base model — use the completions endpoint, not chat)

from openai import OpenAI
# Configured by environment variables
client = OpenAI()

completion = client.completions.create(
    model="JetBrains/Mellum2-12B-A2.5B-Base",
    prompt="def fibonacci(n):\n    ",
    max_tokens=81920,
    temperature=0.6,
    top_p=0.95,
    extra_body={
        "top_k": 20,
    },
)
print("Completion:", completion)

Evaluation

Mellum2 Base pretraining results compared with similarly-sized open base models. All values are self-reported by JetBrains.

For more details, see the Mellum2 Technical Report.

License

Released under the Apache 2.0 license.