Datasetpublic

AnonymizedTMLRSubmission/reasoning-lite

Anonymous Reasoning Lite This repository contains data accompanying an anonymous TMLR submission. It provides 1,211,520 sampled reasoning attempts from four model configurations on competition-math and field-balanced multiple-choice questions. The data omits top-20 alternative-token distributions while retaining realized-token log probabilities and ranks. Contents The same attempts are available in full and metadata-only representations: Configuration group… See the full description on the dataset page: https://huggingface.co/datasets/AnonymizedTMLRSubmission/reasoning-lite.

sourceHugging Facemitupdated 26d agoView on Hugging Face

0likes194downloads

Dataset Card

Anonymous Reasoning Lite

This repository contains data accompanying an anonymous TMLR submission. It provides 1,211,520 sampled reasoning attempts from four model configurations on competition-math and field-balanced multiple-choice questions. The data omits top-20 alternative-token distributions while retaining realized-token log probabilities and ranks.

The same attempts are available in full and metadata-only representations:

Configuration group	Rows	Approximate size	Contents
`meta-math`	59,520	0.44 GiB	Four models and five math splits without token arrays
`meta-gpqa`	1,152,000	1.45 GiB	Four models on multiple-choice questions without token arrays
Four `<model>-math` configs	14,880 each	6.28 GiB total	Math rows with realized-token arrays
Four `<model>-gpqa` configs	288,000 each	27.05 GiB total	Multiple-choice rows with realized-token arrays

text

math/<model>/<task>.parquet
gpqa/<model>/super_gpqa-NNNNN.parquet
meta-math/<model>/<task>.parquet
meta-gpqa/<model>/super_gpqa-NNNNN.parquet

One Parquet row group contains one question's 80 attempts, ordered by seed.

Loading

bash

pip install -U datasets

python

from datasets import load_dataset

math_meta = load_dataset(
    "AnonymizedTMLRSubmission/reasoning-lite",
    "meta-math",
    split="aime_2026",
    streaming=True,
)

gpqa_meta = load_dataset(
    "AnonymizedTMLRSubmission/reasoning-lite",
    "meta-gpqa",
    split="super_gpqa",
    streaming=True,
)

model_data = load_dataset(
    "AnonymizedTMLRSubmission/reasoning-lite",
    "gpt-oss-20b_medium-math",
    split="cmimc_2025",
    streaming=True,
)

Data Format

Field	Description
`task`	Benchmark or split identifier
`model`	Model identifier
`model_key`	Unique configuration key
`data_id`, `seed`	Question identifier and sampling seed
`prompt`, `trigger`	Problem text and generation instruction
`sampling`	Sampling configuration
`text`, `finish_reason`	Generated response and termination reason
`num_prompt_tokens`, `num_completion_tokens`	Token counts
`ground_truth`	Reference answer
`evalscope_extracted_answer`, `evalscope_is_correct`	Extracted answer and rule-based correctness
`cv3b_*`	Three-way verifier outputs and diagnostic values
`llmv_*`	Reference-free verifier criterion scores
`tokens`	Aggregate token statistics and, where present, token-level arrays

Math rows additionally include boxed-answer parsing fields. Multiple-choice rows include stage, full_data_id, uuid, selection_hash, and field taxonomy columns.

cv3b_label is the emitted A, B, or C verifier label. cv3b_prob is the full-vocabulary probability of that emitted label. cv3b_abc_A/B/C is normalized over the three labels. cv3b_ctx_A/B/C contains null-prompt-calibrated diagnostics and should not be interpreted as correctness probabilities.

The reference-free verifier scores problem understanding, reasoning validity, and conclusion support from 1 to 20. Convert an expected criterion score to the retained reward scale with (expected - 1) / 19; average the three criterion rewards for a pointwise score.

The tokens struct contains six aggregate log-probability statistics. Per-model configs also include realized-token, log-probability, and rank arrays. Metadata-only configs omit the arrays.

Reproducibility

Every question has 80 attempts. Use model_key rather than model to distinguish the three reasoning levels of the shared checkpoint. State filtering of aborted or truncated attempts and the verifier score used in any answer-selection analysis.

License

The artifact is provided under the MIT License. Questions and problem statements from the underlying benchmarks remain subject to their applicable terms. The SuperGPQA question content is provided under ODC-BY 1.0 and requires upstream credit.

Upstream Attribution

The multiple-choice questions are derived from the SuperGPQA benchmark, published by the M-A-P Team. Users should credit that benchmark and its paper, "SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines" (2025), when using the question content.

Citation

Citation information will be added after anonymous review.