Modelpublic

Raiff1982/codette-newton-star

sourceHugging Facellama3.1updated 2mo agoView on Hugging Face

0likes6downloads

Model Card

newton-star — ⚠️ Research Artifact: Published Negative Result

Do not use this adapter expecting improvement. It is preserved as the first arm of a controlled self-taught-reasoning (STaR) study, in which it regressed the base adapter from 34.0% to 25.0% (= chance) on GPQA-main.

Trained on 500 of the model's own correct reasoning chains from easy science QA (ARC/OpenBookQA/SciQ, 81% keep yield). The training fit was clean (no overfit); the regression is a property of the method+data, not the optimization: keep-correct STaR on data easier than the evaluation teaches confident easy-mode reasoning that degrades hard-tier performance.

We publish failures with the same rigor as wins — a benchmark that can't be trusted to report failure can't be trusted to report success.

The STaR Study (GPQA-main, reason mode, n=100 per arm)

Adapter	Training data	GPQA	Verdict
newton (untrained baseline)	—	34.0%	reproduced to the decimal, 4 days apart
newton-star-r	350 keep-correct + 180 rationalized	benchmark in progress	complete STaR method
newton-star-hard	350 MMLU-Pro STEM keep-correct	28.0%	attenuated the harm, below baseline
newton-star	500 easy-science keep-correct	25.0%	regressed to chance

Finding: keep-correct self-taught reasoning consolidates existing ability rather than extending it — training data difficulty orders the outcome perfectly, but no keep-correct arm matched the untrained baseline. Full methodology, controls, and changelogs: Codette-Reasoning (see docs/CHANGELOG_2026-07-09.md, docs/CHANGELOG_2026-07-11.md).

Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC