thoughtworks/arithmetic-sorl-data

Arithmetic SoRL Data

Training and evaluation data for the SoRL Arithmetic Interpretability Study.

Small transformers trained on integer addition/subtraction, with SoRL to externalize carry/borrow circuits as explicit abstraction tokens.

Reference: Quirke et al., "Understanding Addition and Subtraction in Transformers" (2024). Paper: arXiv:2402.02619 — see Table 8 for complexity classification and Section 3 for sub-task definitions.

Dataset Structure

Eval Splits

All eval sets use seed=42 for reproducibility. Every model is evaluated on identical examples.

Note: M6 (borrow cascade of length 6) is impossible for 6-digit subtraction — it would require 7 digit positions.

Columns

The eval_stratified split has an additional eval_category column.

Sub-task Labels (Quirke et al. 2024)

Each answer digit requires a specific arithmetic operation. These labels follow Quirke et al. Section 3.

Addition

Subtraction (x >= y)

Complexity Classification (Quirke Table 8)

Complexity = length of longest carry/borrow cascade chain.

Example: 555555+444448=1000003 is S6 — the carry from D0 cascades through 5 consecutive sum-9 positions.

S0: no carries            ~10%
S1: isolated carries      ~50%
S2: cascade of 2          ~26%
S3: cascade of 3           ~9%
S4: cascade of 4           ~3%
S5: cascade of 5           ~1%
S6: cascade of 6          <0.5%

Data Enrichment

Addition: Following Quirke et al., 60% of batches have 40% of digit positions forced to sum-to-9, increasing carry cascade frequency so the model sees enough S4-S6 cases.

Subtraction: 40% of digit positions are forced equal (Dn == D'n), creating borrow propagation cascades (ME/UD). Without this, M3-M5 borrow cascades are extremely rare (M3=0.7%, M4=0.04% in unmodified data). With enrichment: M3=3.0%, M4=0.8%.

Usage

from datasets import load_dataset

ds = load_dataset("thoughtworks/arithmetic-sorl-data", data_dir="add_6digit")
print(ds["train"][0])
# {'tokens': [...], 'labels': ['SA', 'UC', 'US', ...],
#  'complexity': 'S3', 'cascade_depth': 3, ...}

# Stratified eval
eval_ds = load_dataset("thoughtworks/arithmetic-sorl-data",
                       data_dir="add_6digit", data_files="eval_stratified.parquet")

# Addition + subtraction
ds_mixed = load_dataset("thoughtworks/arithmetic-sorl-data", data_dir="add_sub_6digit")

Related

• —Reference paper: Quirke et al., "Understanding Addition and Subtraction in Transformers" (2024)
• —Model checkpoints: thoughtworks/arithmetic-sorl
• —Dashboard: thoughtworks/arithmetic-sorl-dashboard
• —Code: mod_gpt/arithmetic/