Datasetpublic

janewarrenn/CAMP

CAMP: Contextualized Arithmetic with Minimal Pairs CAMP is a benchmark for studying how natural language framing affects LLM accuracy on unit conversion tasks. Each item is a minimal pair: the same underlying computation presented in multiple surface forms, allowing direct comparison of model behavior across prompt conditions while holding the math constant. Key Finding LLMs answer unit conversion questions significantly less accurately when the problem is framed… See the full description on the dataset page: https://huggingface.co/datasets/janewarrenn/CAMP.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

0likes18downloads

Dataset Card

CAMP: Contextualized Arithmetic with Minimal Pairs

CAMP is a benchmark for studying how natural language framing affects LLM accuracy on unit conversion tasks. Each item is a minimal pair: the same underlying computation presented in multiple surface forms, allowing direct comparison of model behavior across prompt conditions while holding the math constant.

Key Finding

LLMs answer unit conversion questions significantly less accurately when the problem is framed in natural language ("Convert 300 km/h to m/s") than when the identical computation is expressed as bare arithmetic ("what is 300 \ 0.277778"*). We find a mean accuracy drop of ~19.9 percentage points across 8 frontier models and 10 domains, despite the underlying math being identical.

Dataset Structure

Columns

Column	Description
`domain`	Conversion domain (e.g., `speed`, `cooking`, `currency`)
`distractor`	Optional substance/item embedded in the prompt (e.g., `water`). Empty string if none.
`number`	The input value to be converted
`answer`	The correct output value (string; timezone answers are formatted times)
`difficulty`	`Easy` (integer inputs) or `Hard` (decimal inputs)
`prompt_natural_language`	Natural language prompt, no reference guide
`prompt_with_guide`	Natural language prompt + conversion reference table
`prompt_math_only`	Bare arithmetic expression (e.g., `what is 300 * 0.277778`). Null for domains where a direct arithmetic expression is not applicable (clothing sizes).

Conditions

Condition	Example
`math_only`	`what is 300 * 0.277778`
`natural_language`	`Convert 300 km/h to m/s. Provide only the numerical value.`
`with_guide`	`Convert 300 km/h to m/s.` + conversion factor table

Domains

15 domains across two groups:

Full (all 3 conditions): bits/bytes, cooking, currency, density, energy, moles/particles, speed, temperature, timezone, volume

Partial (natural language + with\_guide only): clothing sizes (men's pant, men's shoe, women's bra, women's pant, women's shoe)

Distractor Variable

For cooking, volume, density, and moles domains, some prompts embed a specific substance into the question — e.g., "Convert 5 ml of water to tbsp" vs. "Convert 5 ml to tbsp." The distractor field records this substance; rows with an empty distractor are the context-free baseline. Both variants have the same number and answer. No significant accuracy effect was found for this manipulation.

Dataset Size

Stat	Value
Total questions	223,151
Questions with all 3 conditions	42,800
Questions with 2 conditions	180,351
Domains	15

Usage

python

from datasets import load_dataset

ds = load_dataset("janewarren/CAMP", split="test")

# Compare surface forms for one question
row = ds[0]
print(row["prompt_math_only"])
print(row["prompt_natural_language"])
print(row["answer"])

---