ChuGyouk/Qwen3.5-4B-nothink-benchmarks

Qwen3.5-4B (non-thinking) — 13 benchmarks, multi-sample outputs with pass@k

All sampled outputs of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) in non-thinking mode (enable_thinking=False) on 13 benchmarks, with per-response correctness and pass@k / avg@n metrics. One row per problem; every row carries the exact prompt that was sent to the model, all sampled responses, their scores, and the benchmark-level metrics.

Generation setup (identical for every benchmark)

Metrics: avg@n = mean accuracy over the n samples (= pass@1); pass@k = unbiased estimator of Chen et al. 2021, 1 - C(n-c, k)/C(n, k) averaged over problems; maj@n = majority vote over extracted answers (multiple choice only).

Per-benchmark protocol

Multiple-choice instruction (from the Qwen3.5 model card): the prompt is

{question}

A. {option A}
B. {option B}
...

Please show your choice in the `answer` field with only the choice letter, e.g., `"answer": "C"`.

The letter is extracted with, in order: the last "answer": "X" (any quoting/markdown variant), \boxed{X}, answer is X, a final line consisting only of a letter, and the last option X / choice X mention. Responses with no extractable letter (1.4–1.6 %) are scored 0.

HLE judge: Qwen/Qwen3.8-27B served with vLLM (--reasoning-parser qwen3), thinking on, reasoning_effort="xhigh", temperature=1.0, top_p=0.95, top_k=20, min_p=0, presence_penalty=0, max_tokens=65536, structured JSON output (extracted_final_answer, reasoning, correct, confidence) with the verbatim prompt from hle_eval/run_judge_results.py. All 68,896 judgments finished with finish_reason="stop" (no truncation, no errors). Truncated model responses are marked incorrect without judging.

Results

Math (n=32)

General

Additional numbers:

• —bbeh: primary = harmonic mean over the 23 tasks with the paper's +1 adjustment = 20.1; micro average 31.2, macro average 31.6, BBEH-Mini (460) micro average 31.3.
• —ifbench: prompt-level loose 34.4 / strict 28.7; instruction-level loose 37.4 / strict 31.4.
• —hle: exactMatch 5.8, multipleChoice 13.7; RMS calibration error 84.2. Per category: Biology/Medicine 10.9, Chemistry 4.8, Computer Science/AI 4.7, Engineering 5.7, Humanities/Social Science 6.9, Math 9.1, Other 5.0, Physics 5.9.
• —lcb_v6: easy 81.8 / medium 46.1 / hard 15.8; AtCoder 42.1 / LeetCode 39.1; 4.4 % of samples had no code block.
• —gpqa_diamond: Biology 52.3, Chemistry 51.3, Physics 75.1.
• —Model-card numbers for the thinking mode, for reference: MMLU-Pro 79.1, GPQA Diamond 76.2, SuperGPQA 52.9, LiveCodeBench v6 55.8, IFBench 59.2, HMMT Feb 25 74.0.

<details><summary>BBEH per-task avg@32</summary>

</details>

Schema

One row per problem (46,935 rows).

extra_info fields:

from datasets import load_dataset
ds = load_dataset("ChuGyouk/Qwen3.5-4B-nothink-benchmarks", split="test")
aime = ds.filter(lambda r: r["benchmark"] == "aime25")

Notes / caveats

• —Non-thinking mode: no <think> reasoning, but the model still reasons in the visible answer (mean output 1–10K tokens depending on the benchmark).
• —The Qwen3.5 model card lists two different non-thinking sampling recommendations; this run uses the temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5 set.
• —Multiple-choice pass@k with many samples approaches the random-guess ceiling and is reported for completeness only; use avg@n / maj@n.
• —The HLE judge is an open model (Qwen3.8-27B), not the official o3-mini judge, so HLE numbers are not directly comparable to the official leaderboard.
• —Evaluation code: generation via the vLLM OpenAI API, official graders vendored from verl (math_verify), LiveCodeBench, google-deepmind/bbeh and allenai/IFBench.