Datasetpublic

violetxi/single-turn-eval-Qwen3-4B-Instruct-2507-n32

Single-turn eval — Qwen/Qwen3-4B-Instruct-2507 Generated by teaching/inference/single_turn_eval_vllm.py. One row per problem; samples is the list of model responses, scores is per-sample correctness, and mean/best/worst are the aggregates used by mean@N / best@N / worst@N. Eval results (n_samples_per_example = 32) Overall metric value n_examples 1006 mean@32 0.1804 best@32 0.3588 worst@32 0.0537 pass_rate 0.3588… See the full description on the dataset page: https://huggingface.co/datasets/violetxi/single-turn-eval-Qwen3-4B-Instruct-2507-n32.

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes10downloads

Dataset Card

Single-turn eval — `Qwen/Qwen3-4B-Instruct-2507`

Generated by teaching/inference/single_turn_eval_vllm.py. One row per problem; samples is the list of model responses, scores is per-sample correctness, and mean/best/worst are the aggregates used by mean@N / best@N / worst@N.

Eval results (nsamplesper_example = 32)

Overall

metric	value
n_examples	1006
mean@32	0.1804
best@32	0.3588
worst@32	0.0537
pass_rate	0.3588

Per data source

data_source	n	mean@32	best@32	worst@32	pass_rate
CohenQu/POPE-hard-dataset-Qwen3-4B-Instruct-32k-128-filtered-iter3-gemini-success	440	0.0104	0.0773	0.0045	0.0773
Hwilner/imo-answerbench	400	0.3141	0.5725	0.0975	0.5725
MathArena/aime_2026	30	0.5250	0.8000	0.1333	0.8000
MathArena/hmmtnov2025	30	0.4354	0.7333	0.0667	0.7333
d1shs0ap/unified-hard-set-with-student-solutions-guided-rl	76	0.1094	0.4079	0.0263	0.4079
opencompass/AIME2025	30	0.4698	0.7000	0.1667	0.7000

Run config

json

{
  "data_files": [
    "data/instruct/aime-2025/test.parquet",
    "data/instruct/aime-2026/train.parquet",
    "data/instruct/hmmt-nov-2025/train.parquet",
    "data/instruct/imo-answerbench/train.parquet",
    "data/instruct/int_dataset_split/val.parquet",
    "data/instruct/pope_full/test.parquet"
  ],
  "limit": null,
  "output": "teaching/inference/results/single_turn_eval_Qwen3-4B-Instruct-2507_20260505_000244/summary.json",
  "record_log": "teaching/inference/results/single_turn_eval_Qwen3-4B-Instruct-2507_20260505_000244/records.jsonl",
  "solver_model_path": "Qwen/Qwen3-4B-Instruct-2507",
  "solver_max_new_tokens": 16384,
  "temperature": 0.8,
  "top_p": 1.0,
  "n_samples": 32,
  "seed": 0,
  "tensor_parallel_size": 1,
  "data_parallel_size": 8,
  "gpu_memory_utilization": 0.85,
  "max_model_len": 40000,
  "max_num_batched_tokens": 40000,
  "enforce_eager": true,
  "dtype": "bfloat16",
  "enable_thinking": false,
  "hf_dataset_id": null,
  "hf_push": true,
  "hf_private": false,
  "hf_token": null,
  "git_sha": null,
  "wall_time_seconds": 10492.525158166885
}