Modelpublic

ymoslem/ModernBERT-base-AIME-24-25-26-router-qe-binary-vibethinker1.5b-5runs_1eval-10ep-lr5e-05

sourceHugging Facecc-by-nc-sa-4.0updated 22d agoView on Hugging Face

0likes23downloads

Model Card

AIME quality estimator, VibeThinker-1.5B

A binary accept/route classifier for a two-tier LLM cascade on AIME. It reads a VibeThinker-1.5B answer and predicts whether that answer is correct. A route prediction means escalate the query to a stronger model.

Trained for CRE-Router on `ymoslem/AIME-24-25-26-router`.

The generator is named on purpose. This estimator is trained on VibeThinker-1.5B outputs and should be applied to VibeThinker-1.5B outputs. It does not transfer to a different efficient tier without being retrained.

Input format

Build the input exactly as it was trained, or the scores will not mean what they should:

python

text = f"{question} {tokenizer.sep_token} {last_1000_words_of_output} "\
       f"{tokenizer.sep_token} {num_output_tokens}"

Keep only the last 1,000 words of the model's output, then truncate at 4,096 tokens. Class 0 is route, class 1 is accept.

Results

On the 450 rows of test_vibethinker, 90 problems from AIME 2024 to 2026 scored across 5 sampled runs:

Class	F1	Precision	Recall
Route	0.779	0.789	0.769
Accept	0.924	0.920	0.928
Macro	0.851

Accuracy 0.887. It escalates 114 of 450, catching 90 of the 117 wrong answers and wrongly escalating 24 correct ones.

Two caveats worth reading before using these numbers

The evaluation split is also the early-stopping split. This is deliberate, so the checkpoint stays comparable with the earlier estimators in this project, but it means 0.851 is a selection criterion rather than a held-out score.

Much of the signal is output length, and a length rule does as well. The input string ends with the token count, and the predicted accept probability correlates $-0.77$ with it. Composing the actual cascade, VibeThinker-1.5B escalating to Qwen3-30B-A3B-Thinking, and matching the number of escalations:

Escalation rule	Escalated	Cascade accuracy
none, always VibeThinker	0	74.0%
random	115	76.7%
this classifier	114	83.1%
longest answers first	115	84.1%
all, always Qwen3-30B	450	85.8%

Selecting clearly beats not selecting: against random escalation of the same count the classifier gains 6.4 pp. But it does not beat sorting by output length, and the difference between them, $+0.98$ pp for the length rule, has a 95% interval of $[-0.98, +3.47]$ that spans zero. Report the length rule as the baseline, not random.

Training

ModernBERT-base, 10 epochs with early stopping at patience 4, best epoch 3, learning rate 5e-5, effective batch 64 (16 with 4 accumulation steps), class weights for the 86% accept majority, on one A100.

Licence

CC BY-NC-SA 4.0, inherited from the training data, whose AIME 2026 source carries non-commercial share-alike terms. The base model is Apache-2.0.