MU-NLPC/Calc-svamp

Dataset Card for Calc-SVAMP

Summary

The dataset is a collection of simple math word problems focused on arithmetics. It is derived from <https://github.com/arkilpatel/SVAMP/>.

The main addition in this dataset variant is the chain column. It was created by converting the solution to a simple html-like language that can be easily parsed (e.g. by BeautifulSoup). The data contains 3 types of tags:

• —gadget: A tag whose content is intended to be evaluated by calling an external tool (sympy-based calculator in this case)
• —output: An output of the external tool
• —result: The final answer to the mathematical problem (a number)

Supported Tasks

This variant of the dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses. This dataset presents in-context scenarios where models can outsource the computations in the reasoning chain to a calculator.

Construction process

We created the dataset by converting the equation attribute in the original dataset to a sequence (chain) of calculations, with final one being the result to the math problem. We also perform in-dataset and cross-dataset data-leak detection within the Calc-X collection. However, for SVAMP specifically, we detected no data leaks and filtered no data.

Content and data splits

The dataset contains the same data instances as the original dataset except for a correction of inconsistency between equation and answer in one data instance. To the best of our knowledge, the original dataset does not contain an official train-test split. We treat the whole dataset as a testing benchmark.

Attributes:

• —id: problem id from the original dataset
• —question: the question intended to answer
• —chain: series of simple operations (derived from equation) that leads to the solution
• —result: the result (number) as a string
• —result_float: result converted to a floating point
• —equation: a nested expression that evaluates to the correct result
• —problem_type: a category of the problem

Attributes id, question, chain, and result are present in all datasets in Calc-X collection.

Related work

This dataset was created as a part of a larger effort in training models capable of using a calculator during inference, which we call Calcformers.

• —**Calc-X collection** - datasets for training Calcformers
• —**Calcformers collection** - calculator-using models we trained and published on HF
• —**Calc-X and Calcformers paper**
• —**Calc-X and Calcformers repo**

Here are links to the original dataset:

• —**original SVAMP dataset and repo**
• —**original SVAMP paper**

Licence

MIT, consistent with the original source dataset linked above.

Cite

If you use this version of dataset in research, please cite the original SVAMP paper, and Calc-X collection as follows:

@inproceedings{kadlcik-etal-2023-soft,
    title = "Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems",
    author = "Marek Kadlčík and Michal Štefánik and Ondřej Sotolář and Vlastimil Martinek",
    booktitle = "Proceedings of the The 2023 Conference on Empirical Methods in Natural Language Processing: Main track",
    month = dec,
    year = "2023",
    address = "Singapore, Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2305.15017",
}