Datasetpublic

violetxi/single-turn-eval-stage1_proof_pr_delta_variants-n32

Single-turn eval — violetxi/stage1_proof_pr_delta_variants Generated by teaching/inference/single_turn_eval_vllm.py. One row per problem; samples is the list of model responses, scores is per-sample correctness, and mean/best/worst are the aggregates used by mean@N / best@N / worst@N. Eval results (n_samples_per_example = 32) Overall metric value n_examples 1006 mean@32 0.1922 best@32 0.4175 worst@32 0.0457 pass_rate 0.4175… See the full description on the dataset page: https://huggingface.co/datasets/violetxi/single-turn-eval-stage1_proof_pr_delta_variants-n32.

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes19downloads

Dataset Card

Single-turn eval — `violetxi/stage1_proof_pr_delta_variants`

Generated by teaching/inference/single_turn_eval_vllm.py. One row per problem; samples is the list of model responses, scores is per-sample correctness, and mean/best/worst are the aggregates used by mean@N / best@N / worst@N.

Eval results (nsamplesper_example = 32)

Overall

metric	value
n_examples	1006
mean@32	0.1922
best@32	0.4175
worst@32	0.0457
pass_rate	0.4175

Per data source

data_source	n	mean@32	best@32	worst@32	pass_rate
CohenQu/POPE-hard-dataset-Qwen3-4B-Instruct-32k-128-filtered-iter3-gemini-success	440	0.0133	0.1545	0.0023	0.1545
Hwilner/imo-answerbench	400	0.3283	0.5850	0.0775	0.5850
MathArena/aime_2026	30	0.5469	0.8333	0.1333	0.8333
MathArena/hmmtnov2025	30	0.4656	0.8000	0.0667	0.8000
d1shs0ap/unified-hard-set-with-student-solutions-guided-rl	76	0.1435	0.5789	0.0395	0.5789
opencompass/AIME2025	30	0.4979	0.8333	0.1667	0.8333

Run config

json

{
  "data_files": [
    "data/instruct/aime-2025/test.parquet",
    "data/instruct/aime-2026/train.parquet",
    "data/instruct/hmmt-nov-2025/train.parquet",
    "data/instruct/imo-answerbench/train.parquet",
    "data/instruct/int_dataset_split/val.parquet",
    "data/instruct/pope_full/test.parquet"
  ],
  "limit": null,
  "output": "teaching/inference/results/stage1_proof_pr_delta_variants_20260504_171159/summary.json",
  "record_log": "teaching/inference/results/stage1_proof_pr_delta_variants_20260504_171159/records.jsonl",
  "solver_model_path": "violetxi/stage1_proof_pr_delta_variants",
  "solver_max_new_tokens": 16384,
  "temperature": 0.8,
  "top_p": 1.0,
  "n_samples": 32,
  "seed": 0,
  "tensor_parallel_size": 1,
  "data_parallel_size": 5,
  "gpu_memory_utilization": 0.85,
  "max_model_len": 40000,
  "max_num_batched_tokens": 40000,
  "enforce_eager": true,
  "dtype": "bfloat16",
  "enable_thinking": false,
  "hf_dataset_id": null,
  "hf_push": true,
  "hf_private": false,
  "hf_token": null,
  "git_sha": "d6c6facd2310705959045a27e6c7e087bf9361be",
  "wall_time_seconds": 22061.364852190018
}