ygoldi/edumirror-repro-results

EduMirror reproduction — outputs bundle

Full artifacts for the independent reproduction of EduMirror (ICML 2026, arXiv:2606.07948, OpenReview 9CPlmQzUtF).

Layout

• —figures/ — every figure in the logbook (PNG)
• —results/ — raw per-job outputs, exactly as the HF Jobs produced them:
• —judgecheck/, judgecheck_72b/ — judge validity probes (32B fails, 72B passes)
• —claim2_72b/ — kindergarten scalability, 45 episodes, validated 72B judge
• —claim3/ — dual measurement + RSES construct validity, 12 episodes
• —claim4_72b/ — pairwise win-rate heatmap, 120 comparisons, validated judge
• —claim4/ — discarded 32B run (judge failed the probe), kept as evidence
• —claim5/ — intervention strategies, 32 episodes
• —each contains <stage>.json, per-record CSV, run.log, vllm.log, and episodes/*.json (full transcripts + internal states)
• —*.csv — the raw data behind each logbook figure

Read this first

The headline finding is methodological: results/judgecheck/judgecheck.json shows Qwen2.5-32B cannot distinguish a real transcript from one in which every character action was replaced by "performs the required action and awaits the next input" (margin 0.00), and prefers a line-shuffled transcript pairwise (0.375, below chance). Any judged number from that model is noise. judgecheck_72b/ shows Qwen2.5-72B-AWQ passes both probes.

Claims 2 and 4 are therefore reported only from the 72B runs. The 32B claim4/ directory is retained because it appears to confirm the paper (EduMirror first at 0.75) and is a concrete example of how a blind instrument manufactures a false confirmation.

Reproduce

Code: https://huggingface.co/datasets/ygoldi/edumirror-repro-code

hf jobs run --flavor h200 --secrets HF_TOKEN \
  -e EDUMIRROR_MODEL=Qwen/Qwen2.5-72B-Instruct-AWQ -e STAGE=judgecheck \
  -e MAX_MODEL_LEN=32768 --timeout 90m -d vllm/vllm-openai:latest \
  bash -c "curl -sfL https://huggingface.co/datasets/ygoldi/edumirror-repro-code/resolve/main/experiments/job_entrypoint.sh -o /tmp/e.sh && bash /tmp/e.sh"

Swap STAGE for claim2 / claim3 / claim4 / claim5. Always run judgecheck first — if it fails, the other stages' numbers are not interpretable.