cxzsad12e/verl-aha-moment-dataset

Verl Dataset for DeepSeek-R1 "Aha Moment" Reproduction

This dataset is prepared for training with the Verl framework to reproduce the "aha moment" phenomenon observed in DeepSeek-R1-Zero.

What is the "Aha Moment"?

During RL training without any SFT, DeepSeek-R1-Zero spontaneously developed:

• —Self-verification: Checking answers within `` tags
• —Long chain-of-thought: Extended reasoning traces
• —Backtracking: "Wait, that seems wrong..." behavior
• —Metacognition: Allocating more thinking time to hard problems

This emergent behavior is driven by verifiable rewards on reasoning tasks.

Dataset Composition

DeepScaleR (Primary)

• —Source: agentica-org/DeepScaleR-Preview-Dataset (most downloaded DeepScaleR dataset)
• —Train: 36,283 samples
• —Test: 4,032 samples
• —Format: Mathematics problem-answer pairs from AIME, AMC, Omni-MATH, and Still datasets
• —Reward: Rule-based exact match on final answer
• —File: verl_deepscaler.parquet / verl_deepscaler_test.parquet

GSM8K (Grade School Math)

• —Source: openai/gsm8k
• —Train: 7,473 samples
• —Test: 1,319 samples
• —Format: Word problems with numerical answers
• —Reward: Rule-based exact match on final answer

MATH (Competition Math)

• —Source: SuperSecureHuman/competitionmathhfdataset (hendrycks/competitionmath)
• —Train: 7,500 samples
• —Test: 5,000 samples
• —Format: Competition-level math problems (Algebra, Geometry, Number Theory, etc.)
• —Reward: Rule-based exact match on boxed answer

Verl Data Format

Each entry follows Verl's expected schema:

{
    "data_source": "openai/gsm8k",
    "prompt": [{"role": "user", "content": "question text + instruction"}],
    "ability": "math",
    "reward_model": {
        "style": "rule",
        "ground_truth": "numerical_answer"
    },
    "extra_info": {
        "split": "train",
        "index": 0,
        "solution": "full_solution_text"
    }
}

Files

Primary: Parquet Format (Verl Recommended)

• —verl_deepscaler.parquet - Combined training set (GSM8K + MATH)
• —verl_deepscaler_test.parquet - Combined test set
• —train.parquet - Combined training set (same as verl_deepscaler.parquet)
• —test.parquet - Combined test set (same as verldeepscalertest.parquet)
• —train_gsm8k.parquet - GSM8K training split only
• —test_gsm8k.parquet - GSM8K test split only
• —train_math.parquet - MATH training split only
• —test_math.parquet - MATH test split only

Supplementary: Arrow Format (IPC)

• —train.arrow - Combined training set in Arrow IPC format
• —test.arrow - Combined test set in Arrow IPC format
• —train_gsm8k.arrow - GSM8K training split in Arrow IPC format
• —test_gsm8k.arrow - GSM8K test split in Arrow IPC format
• —train_math.arrow - MATH training split in Arrow IPC format
• —test_math.arrow - MATH test split in Arrow IPC format

Usage with Verl

1. 1.Copy the parquet files to your HDFS or local data directory
2. 2.In your Verl config, set data.path to the directory containing these files
3. 3.The dataset will be automatically loaded with the chat template applied

Reward Function

For these datasets, use a rule-based reward function that:

1. 1.Extracts the final answer from the model's response
2. 2.Compares it against ground_truth
3. 3.Returns 1.0 for exact match, 0.0 otherwise

Tips for Reproducing the "Aha Moment"

1. 1.Start from a base model (not an instruct model) - the "aha moment" is about discovering reasoning from scratch
2. 2.Use pure RL (no SFT first) - this is how R1-Zero achieved the phenomenon
3. 3.Large batch sizes - more diverse rollouts help the model explore
4. 4.Verifiable rewards - exact match on answers gives clear signal
5. 5.Monitor response length - the model should start producing longer, more structured responses
6. 6.Look for self-correction patterns - "Wait, let me check..." type behaviors