AIML-TUDA/SLR-Bench-German

<div style="display: flex; justify-content: flex-start;"><img src="https://raw.githubusercontent.com/ml-research/ScalableLogicalReasoning/master/images/SLR-Bench2.jpg" alt="Preview" style="width: 40vw; min-width: 300px; max-width: 600px;"> </div>

Dataset Description

• —Language(s) (NLP): English
• —Point of Contact: Lukas Helff
• —License: CC BY

🧠 SLR-Bench-German: Scalable Logical Reasoning Benchmark (German Edition)

  

SLR-Bench Multilingual Versions:

      

SLR-Bench-German is the German-language pendant of the original **SLR-Bench** dataset. It follows the same symbolic structure, evaluation framework, and curriculum as the English version but provides all natural-language task prompts translated into German.

This enables systematic evaluation and training of Large Language Models (LLMs) in logical reasoning in german, supporting both multilingual reasoning and cross-lingual generalization research.

DS Overview

• —Curriculum: 20 complexity levels, grouped into 4 broad tiers (basic, easy, medium, hard)
• —Tasks: >19,000, each comprising: A natural language prompt, an executable validation program for automatic evaluation, and a latent ground-truth rule.
• —Application: SLR-Bench can used to evaluate conventional and reasoning LLMs (e.g., GPT-4o, Llama-3, Gemini, DeepSeek-R1) and to train models via curriculum learning.

Key Features of SLR

• —🔨 Automatic Task Generation: Synthesize new inductive reasoning tasks with controllable complexity, novel logic rules, and natural language prompts—no need for human annotation.
• —🧩 Programmable & Scalable: Specify your own logic vocabulary, grammar, rule distributions, and task parameters; supports curriculum-style scaling and out-of-distribution task creation.
• —🧠 Symbolic, Automated Evaluation: Deterministically verify LLM outputs via the validation program, not MCQA, LLM judge, or exact matching.
• —📈 Curriculum Learning: Use SLR-Bench, a structured 20-level benchmark, for evaluating and training models across a span of logical challenges.

Quick Start

Loading the Dataset

from datasets import load_dataset
# Load SLR-Bench test split
ds = load_dataset("AIML-TUDA/SLR-Bench-German", "v1-All", split="test")

Evaluate using SLR-Bench

Requires the `evaluate` library and a Prolog interpreter installed on your system (e.g., SWI-Prolog). Install the required dependencies via:

pip install evaluate
sudo apt-get install swi-prolog

Example Usage

from evaluate import load
symbolic_judge = load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")
rules = ds["ground-truth rule"]  # For demo only—use model predictions in practice
references = [
    {
        "validation_program": p,
        "evaluation_config": {
            "positive_predicate": "ost",
            "negative_predicate": "west"
        }
    } for p in ds["validation program"]
]

results = symbolic_judge.compute(predictions=rules, references=references)
print(results)

Note: For real evaluation, replace `rules` with your model's predicted rules. Here, we use ground-truth rules for demonstration only.

Example results:

{'accuracy': 1.0,
 'partial_score': 1.0,
 'syntax_score': 1.0,
 'detailed_results': [{'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.014362812042236328},
                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012364625930786133}]
}

Dataset Columns

SLR-Bench Curriculum

SLR-Bench Curriculum: level-wise configurations, detailing language and task parameters for each difficulty stage. Language complexity is systematically increased by expanding the number of car constants and predicates. Task configuration grows via adapting problem size, background sampling, rule length, and rule sampling strategy. The final column reports the approximate combinatorial size of unique tasks available at each level.

Licensing Information

SLR-Bench is made available under the CC BY license.

Citation

If you use this dataset or framework, please cite:

@incollection{helff2025slrautomatedsynthesisscalable,
      title={SLR: Automated Synthesis for Scalable Logical Reasoning}, 
      author={Lukas Helff and Ahmad Omar and Felix Friedrich and Antonia Wüst and Hikaru Shindo and Rupert Mitchell and Tim Woydt and Patrick Schramowski and Wolfgang Stammer and Kristian Kersting},
      year={2025},
      booktitle ={Working Notes of the NeurIPS Workshop on Foundations of Reasoning in Language Models},
      url={https://arxiv.org/abs/2506.15787}, 
}