Datasetpublic

argo11/japanese-math-empirical-difficulty-pilot-50k

Japanese Math Empirical Difficulty Pilot 50k This dataset is a 50,000-problem empirical difficulty pilot, not a full empirical labeling of the original 5.66M-row source dataset. It was created for LLM-jp experiment 0399, Team Victory SFT, to validate empirical difficulty label distribution, downstream split behavior, and the rollout/scoring pipeline before attempting labeling at the full 5.6M scale. Current Status This upload uses the v3 scorer with assistant-only… See the full description on the dataset page: https://huggingface.co/datasets/argo11/japanese-math-empirical-difficulty-pilot-50k.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes40downloads

Dataset Card

Japanese Math Empirical Difficulty Pilot 50k

This dataset is a 50,000-problem empirical difficulty pilot, not a full empirical labeling of the original 5.66M-row source dataset.

It was created for LLM-jp experiment 0399, Team Victory SFT, to validate empirical difficulty label distribution, downstream split behavior, and the rollout/scoring pipeline before attempting labeling at the full 5.6M scale.

Current Status

This upload uses the v3 scorer with assistant-only extraction plus additional equivalence fallbacks for notation, algebraic equivalence, solution sets, no-solution synonyms, and numeric approximations. Earlier v1/v2 uploads are superseded for empirical frontier labeling.

Source Dataset

—Source dataset: HayatoHongoEveryonesAI/qa_verify_cot_new_6M_v6
—Hugging Face dataset viewer reports the source as default · 5.66M rows.
—Local preprocessing manifest saw 5663154 source rows and produced TV_valid_clean.parquet with 5461079 clean rows.
—This pilot samples 50,000 problems from TV_valid_clean.parquet.
—Source dataset license: not specified in the source dataset card at upload time. Inherited source constraints apply.

Experiment

—LLM-jp experiment issue: https://github.com/llm-jp/experiments/issues/399
—Assigned issue URL provided for this upload: https://github.com/issues/assigned?issue=llm-jp%7Cexperiments%7C399
—Experiment directory: /groups/gcg51557/experiments/0399_tv_sft

Files and Configs

Use separate configs because the two parquet files have different schemas:

python

from datasets import load_dataset

problem_difficulty = load_dataset("argo11/japanese-math-empirical-difficulty-pilot-50k", "problem_difficulty")
rollout_scored = load_dataset("argo11/japanese-math-empirical-difficulty-pilot-50k", "rollout_scored")

—problem_difficulty: one row per problem with empirical correctness counts and bucket flags.
—rollout_scored: one row per rollout completion with extracted answer and scoring result.
—metadata/: scoring summary, scoring manifest, and source preprocessing manifest.

Rollout and Scoring

—Problems: 50,000
—Rollout completions: 400,000
—Completions per problem: 8
—Rollout model: llm-jp/llm-jp-4-8b-thinking
—Scoring: Math-Verify answer verification against expected_answer
—Scorer version: v3
—Mean correct rate: 0.9414375

Empirical Difficulty Labels

correct_count is the number of correct completions out of 8.

text

is_emp_frontier: correct_count == 0
is_emp_learnable: 1 <= correct_count <= 6
is_emp_easy: correct_count >= 7

[image]

Correct-count distribution:

text

Empirical buckets:

text

frontier: 786
learnable: 3,868
easy: 45,346

Limitations

—This is a 50k pilot, not the full 5.66M-row source dataset.
—The pilot distribution should not be treated as a direct estimate of the full source dataset difficulty distribution.
—Correctness depends on answer extraction and Math-Verify behavior; some mathematically equivalent answers may still be missed.
—Rollout scoring reflects the specific model, prompt format, sampling settings, and scoring code recorded in metadata/.