Huangsssz/SmolLM3-3B-MOPD-Top64-DT-NoThinking-500

SmolLM3-3B-MOPD-Top64-DT-NoThinking-500

Research checkpoint after 500 optimizer updates, using teacher/student Top64-token-intersection distillation with domain_token aggregation. Student generation and all teacher-scoring prompts used thinking OFF. This is a full BF16 Hugging Face model export, with its tokenizer and original chat template.

Training

• —Initial student: BytedTsinghua-SIA/Open-MOPD-SmolLM3-3B-MixSFT, revision c9e7bad031667828656ead188d7e8ea162c048a4.
• —Frozen domain-specific teachers: Math, Code, Instruction following. Each answer is scored by its matching teacher.
• —Optimizer: adam, learning rate 2.5e-7, weight decay 0, gradient clipping 1.
• —Aggregation: domain_token; 48 student responses per update, 16 per domain.
• —Maximum training answer length: 4096 tokens; temperature 1, top-p 1, top-k -1, seed 42.
• —BF16 model computation, FP32 master parameters and gradient accumulation.
• —Training code derives from opd_geometry. Revisions and selected configuration are in training_config.json.

Adam settings: beta1=0.9, beta2=0.98, epsilon=1e-8. The Top64 candidates are the intersection of teacher and student top-64 token sets.

Completed evaluation snapshot

All scores below use thinking OFF. GPQA averages four answers; AIME reports pass@8 separately by year. AIME belongs to mathematics but is reported as a separate benchmark category.

These are single fixed-protocol measurements. AIME has only 30 questions per year; small score differences should not be interpreted as established statistical improvements. Pending evaluations will require a later metadata update.

Inference with thinking disabled

The tokenizer supports both modes; downloading this checkpoint does not force every caller into no-thinking mode. Pass enable_thinking=False explicitly, and use a system prompt that does not request <think>...</think> output.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "Huangsssz/SmolLM3-3B-MOPD-Top64-DT-NoThinking-500"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "Solve 2x + 3 = 11."}]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    enable_thinking=False, return_tensors="pt", return_dict=True,
).to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=1024, do_sample=False,
        eos_token_id=128012, pad_token_id=128012,
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use a Transformers release with SmolLM3 support. The validated local runtime used Transformers 5.12.1. The short deterministic example above is a usage example, not the benchmark decoding protocol.

Files and attribution

SHA256SUMS and export_manifest.json describe the exported model files. Optimizer states and native distributed training checkpoints are retained on the cluster and are not part of this inference repository. The base model and teacher families are credited above; the upstream MixSFT model card declares Apache-2.0. This export preserves the model configuration, including its RoPE settings and stop-token behavior.