sungyub/ifbench-verl

IFBench-VERL: Instruction Following Evaluation Dataset for VERL Training

Overview

IFBench-VERL is a comprehensive instruction-following evaluation dataset formatted for VERL (Versatile Reinforcement Learning) training pipelines. This dataset contains 95,373 high-quality examples with 54 different constraint types, enabling systematic training and evaluation of instruction-following capabilities in language models.

The dataset is converted from allenai/IF_multi_constraints_upto5 into VERL-compatible format with integrated IFEval-based reward modeling for automatic constraint verification.

Key Statistics

• —Total Examples: 95,373
• —Format: Parquet (VERL-compatible)
• —Data Source: allenai/IF_multi_constraints_upto5
• —License: ODC-BY-1.0 (Open Data Commons Attribution License)
• —Conversion Date: November 3, 2025
• —Constraint Types: 54 different instruction-following constraints
• —Constraint Categories: 13 (keywords, language, length, content, format, punctuation, etc.)
• —Max Constraints per Example: Up to 5 constraints
• —Reward Model: IFEval-based automatic evaluation

Dataset Structure

Schema

The dataset follows the VERL standard schema format:

{
    'data_source': 'allenai/IF_multi_constraints_upto5',
    'prompt': [
        {
            'role': 'system',
            'content': 'You are a helpful assistant...'
        },
        {
            'role': 'user',
            'content': 'Write a paragraph about...'
        }
    ],
    'ability': 'instruction_following',
    'reward_model': {
        'style': 'ifeval',
        'ground_truth': '[{"instruction_id": ["keywords:existence"], "kwargs": [{"keywords": ["example", "test"]}]}]'
    },
    'extra_info': {
        'index': 0
    },
    'dataset': 'ifeval'
}

Field Descriptions

Instruction Following Tasks

This dataset tests 54 different constraint types across 13 major categories:

Constraint Categories

1. 1.Keywords (8 constraints)
2. 2.keywords:existence - Require specific keywords
3. 3.keywords:frequency - Control keyword frequency
4. 4.keywords:forbidden_words - Prohibit certain words
5. 5.keywords:letter_frequency - Control letter frequency
6. 6.keywords:no_adjacent_consecutive - Prevent adjacent letters
7. 7.keywords:word_once - Require words exactly once
8. 8.keywords:palindrome - Require palindrome words
9. 9.keywords:keyword_specific_position - Place keywords at specific positions
10. 10.keywords:start_end - Require start/end with specific text

1. 1.Language (1 constraint)
2. 2.language:response_language - Require response in specific language

1. 1.Length Constraints (4 constraints)
2. 2.length_constraints:number_sentences - Control sentence count
3. 3.length_constraints:number_paragraphs - Control paragraph count
4. 4.length_constraints:number_words - Control word count
5. 5.length_constraints:nth_paragraph_first_word - Control specific paragraph's first word

1. 1.Detectable Content (2 constraints)
2. 2.detectable_content:number_placeholders - Require placeholders (e.g., [NAME])
3. 3.detectable_content:postscript - Require postscript section

1. 1.Detectable Format (8 constraints)
2. 2.detectable_format:number_bullet_lists - Require bullet lists
3. 3.detectable_format:constrained_response - Constrain response format
4. 4.detectable_format:number_highlighted_sections - Require highlighted sections
5. 5.detectable_format:multiple_sections - Require multiple sections
6. 6.detectable_format:json_format - Require JSON format
7. 7.detectable_format:title - Require title
8. 8.detectable_format:sentence_hyphens - Require sentence with hyphens
9. 9.detectable_format:square_brackets - Require square brackets
10. 10.detectable_format:bigram_wrapping - Require bigram wrapping

1. 1.Combination (2 constraints)
2. 2.combination:two_responses - Require two separate responses
3. 3.combination:repeat_prompt - Repeat prompt then answer

1. 1.Start/End (2 constraints)
2. 2.startend:end_checker - Control response ending
3. 3.startend:quotation - Wrap response in quotation marks

1. 1.Change Case (3 constraints)
2. 2.change_case:capital_word_frequency - Control capitalized word frequency
3. 3.change_case:english_capital - All uppercase
4. 4.change_case:english_lowercase - All lowercase

1. 1.Punctuation (4 constraints)
2. 2.punctuation:no_comma - Prohibit commas
3. 3.punctuation:punctuation_dot - End sentences with periods
4. 4.punctuation:punctuation_exclamation - Use exclamation marks

1. 1.Copy (4 constraints)
2. 2.copy:repeat_phrase - Repeat specific phrase
3. 3.copy:copy - Copy from prompt
4. 4.copy:copying_simple - Simple copying task
5. 5.copy:copying_multiple - Multiple copying operations

1. 1.Paragraphs (2 constraints)
2. 2.paragraphs:paragraphs - Basic paragraph structure
3. 3.paragraphs:paragraphs2 - Alternative paragraph structure

1. 1.First/Last Word (4 constraints)
2. 2.first_word:first_word_sent - Control first word of sentence
3. 3.first_word:first_word_answer - Control first word of answer
4. 4.last_word:last_word_sent - Control last word of sentence
5. 5.last_word:last_word_answer - Control last word of answer

1. 1.Count (6 constraints)
2. 2.count:lowercase_counting - Count lowercase letters
3. 3.count:counting_composition - Counting composition
4. 4.count:count_unique - Count unique elements
5. 5.count:count_increment_word - Count with incremental words
6. 6.letters:letter_counting - Count specific letters
7. 7.letters:letter_counting2 - Alternative letter counting

Reward Model

IFEval-Based Automatic Evaluation

The dataset uses IFEval (Instruction Following Evaluation) for automatic constraint verification. Each example includes:

• —Constraint Specification: JSON-encoded list of constraints with parameters
• —Automatic Checking: Self-contained scorer in the datatrove library
• —Binary Scoring: Each constraint is either satisfied (1.0) or not (0.0)
• —Average Score: Final score is the average across all constraints

Ground Truth Format

The reward_model.ground_truth field contains JSON-encoded constraint specifications:

[
  {
    "instruction_id": [
      "keywords:existence",
      "length_constraints:number_words"
    ],
    "kwargs": [
      {"keywords": ["example", "test"]},
      {"num_words": 100, "relation": "at least"}
    ]
  }
]

Self-Contained Scorer

The IFEval scorer is included in the datatrove library at datatrove.utils.reward_score.ifeval:

from datatrove.utils.reward_score.ifeval import compute_score

# Evaluate model output against constraints
result = compute_score(
    model_output="Your model's response...",
    ground_truth='[{"instruction_id": [...], "kwargs": [...]}]'
)

# Result format
{
    "score": 0.85,  # Average success rate across all constraints
    "reward_fmt": 1.0,
    "reward_think": 1.0
}

Data Access

Using HuggingFace Datasets

from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("sungyub/ifbench-verl", split="train")

# Example usage
example = dataset[0]
print(f"Prompt: {example['prompt']}")
print(f"Constraints: {example['reward_model']['ground_truth']}")

Using Pandas

import pandas as pd

# Load as pandas DataFrame
df = pd.read_parquet("hf://datasets/sungyub/ifbench-verl/train-00000-of-00001.parquet")

# View statistics
print(df.info())
print(df.head())

Using Polars

import polars as pl

# Load with Polars for faster processing
df = pl.read_parquet("hf://datasets/sungyub/ifbench-verl/train-00000-of-00001.parquet")

print(df.describe())

Technical Details

Metadata Standards

This dataset includes metadata following:

• —ML Commons Croissant 1.1: Structured dataset metadata
• —VERL Format: Compatible with VERL training pipelines
• —IFEval Integration: Self-contained constraint checking

Supported Libraries

• —datasets (HuggingFace)
• —pandas
• —polars
• —dask

Dependencies for Reward Scoring

To use the IFEval reward scorer, install:

pip install datatrove[reward_scoring]

# Or install specific dependencies:
pip install langdetect immutabledict nltk

Use Cases

1. Instruction Following Training

Train language models to better follow complex instructions with multiple constraints:

from datasets import load_dataset

dataset = load_dataset("sungyub/ifbench-verl", split="train")

# Use for supervised fine-tuning or RL training
for example in dataset:
    prompt = example['prompt']
    constraints = example['reward_model']['ground_truth']
    # Train your model...

2. VERL/GRPO Training

Direct integration with VERL training pipelines:

# The 'dataset' field enables automatic verifier routing in GRPO
# The reward model automatically uses IFEval scoring

3. Benchmark Evaluation

Evaluate model instruction-following capabilities across 54 constraint types:

from datatrove.utils.reward_score.ifeval import compute_score

# Evaluate model output
score = compute_score(
    model_output=model_response,
    ground_truth=example['reward_model']['ground_truth']
)

print(f"Instruction following score: {score['score']:.2%}")

4. Constraint Analysis

Analyze which constraint types are most challenging:

import json

# Group by constraint types
constraint_stats = {}
for example in dataset:
    gt = json.loads(example['reward_model']['ground_truth'])
    for constraint_id in gt[0]['instruction_id']:
        category = constraint_id.split(':')[0]
        constraint_stats[category] = constraint_stats.get(category, 0) + 1

print(constraint_stats)

Citation and License

License

This dataset is licensed under ODC-BY-1.0 (Open Data Commons Attribution License), following the original dataset's license terms. You are free to:

• —Share and redistribute the dataset
• —Adapt and modify the dataset
• —Use the dataset commercially

Under the condition of:

• —Attribution: You must give appropriate credit to the original dataset creators

Citation

If you use this dataset, please cite the original IF dataset:

@misc{if_multi_constraints_upto5,
  title={IF Multi-Constraints (Up to 5)},
  author={AI2},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/allenai/IF_multi_constraints_upto5},
  year={2024}
}

And this VERL-formatted version:

@misc{ifbench_verl,
  title={IFBench-VERL: Instruction Following Evaluation Dataset for VERL Training},
  author={Sungyub Kim},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/sungyub/ifbench-verl},
  year={2025}
}

Responsible Use

This dataset is intended for research and educational purposes in accordance with:

• —AI2's Responsible Use Guidelines
• —Open Data Commons Attribution License terms

Please ensure ethical use of this dataset and be mindful of potential biases in instruction-following evaluation.

Additional Resources

• —Original Dataset: allenai/IF_multi_constraints_upto5
• —IFEval Paper: Instruction-Following Evaluation
• —VERL Documentation: VERL Training Guide
• —Datatrove Library: HuggingFace Datatrove

Contact

For questions or issues regarding this dataset conversion, please open an issue on the dataset repository.