Datasetpublic

neurallambda/arithmetic_dataset

Arithmetic Puzzles Dataset A collection of arithmetic puzzles with heavy use of variable assignment. Current LLMs struggle with variable indirection/multi-hop reasoning, this should be a tough test for them. Inputs are a list of strings representing variable assignments (c=a+b), and the output is the integer answer. Outputs are filtered to be between [-100, 100], and self-reference/looped dependencies are forbidden. Splits are named like: train_N 8k total examples of puzzles… See the full description on the dataset page: https://huggingface.co/datasets/neurallambda/arithmetic_dataset.

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes115downloads

Dataset Card

Arithmetic Puzzles Dataset

A collection of arithmetic puzzles with heavy use of variable assignment. Current LLMs struggle with variable indirection/multi-hop reasoning, this should be a tough test for them.

Inputs are a list of strings representing variable assignments (c=a+b), and the output is the integer answer.

Outputs are filtered to be between [-100, 100], and self-reference/looped dependencies are forbidden.

Splits are named like:

—train_N 8k total examples of puzzles with N variables
—test_N 2k more examples with N variables

Train/test leakage is prevented: all training examples are filtered out of the test set.

Conceptually the data looks like this:

python

Input:

  a=1
  b=2
  c=a+b
  solve(c)=

Output:
  3

In actuality it looks like this:

python

{
    "input": ['var_0=1', 'var_1=2', 'var_2=a+b', 'solve(var_2)='],
    "output": 3
}

Loading the Dataset

python

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("neurallambda/arithmetic_dataset")

# Load specific splits
train_small = load_dataset("neurallambda/arithmetic_dataset", split="train_10")
test_small = load_dataset("neurallambda/arithmetic_dataset", split="test_10")

Preparing Inputs

To prepare the inputs as concatenated strings, you can do this:

python

def prepare_input(example):
    return {
        "input_text": "
".join(example["input"]),
        "output": example["output"]
    }

# Apply the preparation to a specific split
train_small_prepared = train_small.map(prepare_input)

# Example of using the prepared dataset
for example in train_small_prepared.select(range(5)):  # Show first 5 examples
    print("Input:", example["input_text"])
    print("Output:", example["output"])
    print()

This will produce output similar to:

Input: var_0=5
var_1=2
var_2=-2 + -8
var_3=3
var_4=4
var_5=var_2
var_6=var_3 * 10
var_7=var_2 - var_0
var_8=var_1
var_9=-2 - 9
solve(var_3)=
Output: 3