CoolFace
Datasetpublic

jgallego9/flexibility-trap-repro-results

The Flexibility Trap — reproduction results Raw and summarized outputs for a scaled reproduction of arXiv:2601.15165 / OpenReview kpgURPRMGf. Result folders claims-1-3/claims-1-3-h200-p8-s4-g64-r7-local-download: 8 evenly spaced GSM8K problems, 4 stochastic samples per AO/AR order, 64 generated tokens, 41 paired logical-fork events. claims-4-5/claims-4-5-h200-p12-b4-g256-r7-local-download: 12 evenly spaced GSM8K problems, base and released JustGRPO checkpoint… See the full description on the dataset page: https://huggingface.co/datasets/jgallego9/flexibility-trap-repro-results.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes37downloads
Dataset Card

The Flexibility Trap — reproduction results

Raw and summarized outputs for a scaled reproduction of arXiv:2601.15165 / OpenReview kpgURPRMGf.

Result folders

  • —claims-1-3/claims-1-3-h200-p8-s4-g64-r7-local-download: 8 evenly spaced GSM8K problems, 4 stochastic samples per AO/AR order, 64 generated tokens, 41 paired logical-fork events.
  • —claims-4-5/claims-4-5-h200-p12-b4-g256-r7-local-download: 12 evenly spaced GSM8K problems, base and released JustGRPO checkpoint, 1/2/4/8 confidence-ranked tokens per call, 256 generated tokens.

Every folder contains run_config.json, summary.json, raw JSONL, CSV metrics, and standalone Plotly HTML.

Validated headline results

  • —Claim 1: not reproduced at this scale; AO Pass@1/2/4 = 0.750/0.875/1.000, AR = 0.781/0.875/1.000.
  • —Claim 2: compatible but non-discriminative; all 8 problems were solved by both orders.
  • —Claim 3: reproduced in mean; AR-counterfactual minus AO entropy = +0.229 nats, bootstrap 95% CI [0.054, 0.402], n=41.
  • —Claim 4: released checkpoint 11/12 = 91.7% at strict one-token decoding; 89.1% lies inside the Wilson interval. Training was not repeated.
  • —Claim 5: proxy reproduced; JustGRPO-minus-base gain rose from 0.0 points at 1 token/call to +58.3 points at 8 tokens/call.

Provenance

  • —Claims 1–3 H200 job: https://huggingface.co/jobs/jgallego9/6a57c4b2b1669a49bf075a9c
  • —Claims 4–5 H200 job: https://huggingface.co/jobs/jgallego9/6a57c4bc85d9643ce16d5314
  • —Base checkpoint: GSAI-ML/LLaDA-8B-Instruct
  • —Tuned checkpoint: nzl-thu/LLaDA-Instruct-JustGRPO-GSM8K
  • —Official-code audit commit: 1a2fddb5c6655597e63081c0af5ebb718a849f39
  • —Independent validation: 64/64 and 96/96 expected raw rows; both summaries recomputed exactly.

Download

bash
hf download jgallego9/flexibility-trap-repro-results --repo-type dataset --local-dir flexibility-trap-results

See the published Trackio logbook for claim-by-claim interpretation, limitations, costs, poster, and artifact bundle.