Modelpublic

Tuwhy/Olmo-3-7B-Think-OPSA

sourceHugging Faceapache-2.0updated 8d agoView on Hugging Face

1likes425downloads

Model Card

Olmo-3-7B-Think-OPSA

This repository contains the step 19 checkpoint of Olmo-3-7B-Think trained with On-Policy Self-Adaptation (OPSA).

![Paper](https://arxiv.org/pdf/2608.31046) ![Code](https://github.com/DripNowhy/On-Policy-Self-Adaptation) ![Blog](https://dripnowhy.github.io/On-Policy-Self-Adaptation/) ![Checkpoints](https://huggingface.co/collections/Tuwhy/on-policy-self-adaptation) ![W&B Logs](https://wandb.ai/whywhyyy0731-purdue-university/opsa-public/workspace?nw=nwuserwhywhyyy0731)

About OPSA

OPSA improves a policy without a teacher model, reward model, reference-model forward pass, or task reward. The canonical setup trains the 20% of valid response tokens with the lowest actor log probabilities and assigns entropy-adaptive negative advantages between -0.5 and -1.0.

This checkpoint was trained using only the questions from DAPO-17k.

—Base model: allenai/Olmo-3-7B-Think
—Training checkpoint: step 19 (step019)
—Architecture: Olmo3ForCausalLM
—Weight dtype: bfloat16
—Context window: 65,536 tokens

Evaluation

The three math columns report Avg@32 / Pass@32; the two out-of-distribution columns report Avg@32. All scores are percentages; absolute gains are percentage points.

Variant	AIME24	AIME25	HMMT25	MBPP+	GPQA_D
Olmo3-7B-TK Base	71.88 / 93.33	61.77 / 90.00	41.98 / 73.33	64.55	38.86
w/ OPSA (step 19)	77.29 / 93.33	67.08 / 93.33	47.50 / 83.33	66.33	40.93
Δ	+5.41 / +0.00	+5.31 / +3.33	+5.52 / +10.00	+1.78	+2.07
Relative gain	+7.5% / +0.0%	+8.6% / +3.7%	+13.1% / +13.6%	+2.8%	+5.3%

Results and gains above are reproduced from the authors' reported results table. GPQA_D denotes GPQA-Diamond. See the paper and project repository for experimental details.

Usage

Recommended sampling settings

After OPSA training, this model may benefit from a higher sampling temperature. We recommend trying `temperature=1.5` with do_sample=True, top_p=0.8, and top_k=20 as a starting point for improved reasoning performance. The best temperature depends on the task, so tune it on your own validation examples; higher temperatures do not always yield better results.

Quick start

Use the checkpoint's chat template to format prompts for the Think model.

bash

pip install "transformers>=4.57.0" accelerate torch

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Tuwhy/Olmo-3-7B-Think-OPSA"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Solve: 3x + 5 = 20."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

outputs = model.generate(
    **inputs,
    do_sample=True,
    temperature=1.5,
    top_p=0.8,
    top_k=20,
    max_new_tokens=32768,
)
print(tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
))

The decoding settings above are an inference example, not a specification of the evaluation protocol for every result in the table.

License and limitations

This checkpoint is released under the Apache 2.0 license, following the base model. It can produce incorrect, biased, or harmful content; OPSA's reasoning results do not establish improved safety. For the original model's development details and limitations, see the base model card.

Citation

OPSA

If you find this work useful, please cite:

bibtex

@article{ding2026does,
  title={Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement},
  author={Ding, Yi and Zhang, Ruqi},
  journal={arXiv preprint arXiv:2608.31046},
  year={2026}
}

Base model

Please also refer to the Olmo 3 model card for the original model citation.