CoolFace
Datasetpublic

duoduoyeah/gsm8k-promise

GSM8K-Promise A transform of openai/gsm8k (the main subset) into a "promise" format: the original chain-of-thought is kept in answer, and an added promise_answer rewrites the solution so the model emits only operations and operands — every computed value is a var that an external tool resolves. The model itself never computes, compares, or rounds, so the reasoning is machine-translation-safe (numbers don't get mangled in translation) and tool-verifiable. train: 7,473 rows ·… See the full description on the dataset page: https://huggingface.co/datasets/duoduoyeah/gsm8k-promise.

sourceHugging Facemitupdated 3mo agoView on Hugging Face
0likes35downloads
Dataset Card

GSM8K-Promise

A transform of openai/gsm8k (the main subset) into a "promise" format: the original chain-of-thought is kept in answer, and an added promise_answer rewrites the solution so the model emits only operations and operands — every computed value is a var that an external tool resolves. The model itself never computes, compares, or rounds, so the reasoning is machine-translation-safe (numbers don't get mangled in translation) and tool-verifiable.

  • —train: 7,473 rows · test: 1,319 rows (same split sizes as GSM8K).
  • —Every row's promise_answer resolves, via the tool semantics below, to the exact original #### answer.

Format

In promise_answer, the k-th <<expr>> call binds vark (var1, var2, …) written immediately after it, e.g. Mimi has 2 x 12 = <<2*12>>var1 sea shells. Once bound, a value is referenced only by its var name. There is no =result and no >/< inside <<>>. promise_bindings maps each var to its expression.

What the model may predict. Two rules govern the bare operands it can write; everything computed must be a var referenced by name:

  1. 1.A number that appears verbatim in the question.
  2. 2.A well-known constant (e.g. 100 for percent, 60 min/hr, 12 per dozen, 7 days/week).

The permitted operations are exactly: arithmetic + - * / // % **, and the tool calls solve, compare/select, and log. Per row, allowed_literals enumerates the bare numbers allowed under rules 1–2.

Tool semantics (how to resolve a row):

  • —arithmetic: + - * / // % ** over the bound vars and allowed_literals.
  • —solve([lhs, rhs], …, target) — solve a system of linear equations for target; unknowns are bare letters (x, c, …). E.g. <<solve([c + d, 21], [d, 2*c], c)>>var1.
  • —compare(a, b) → "gt"|"eq"|"lt"; select(key, {gt:…, eq:…, lt:…}) picks a value. Used together for any extremum/decision (branch rows) so the model never asserts which value is larger.
  • —log(x, base) — closed form for iterate-until-stop (×k each step) rows.

Fields

fieldtypedescription
idstringgsm8k_<split>_<source_idx>
source_split / source_idxstring / intprovenance in the seed-42-shuffled GSM8K split
questionstringoriginal GSM8K question
answerstringoriginal GSM8K chain-of-thought + #### answer
promise_answerstringpromise-format solution (see above)
promise_bindingsstringJSON object {var: expr} (string because var keys vary)
allowed_literalslist[string]bare numbers permitted in promise_answer (question numbers + well-known constants)
transform_statusstringok / edited / branch / skipped
transform_notesstringrationale for non-ok rows

transform_status: ok = pure var-masking that round-trips to the original; edited = sanctioned changes (added a missing <<>>, decimal-percent → /100, equation solved via solve(), etc.); branch = uses compare/select; skipped = no faithful transform (the source CoT contradicts its own #### answer, or the math is non-linear / not tool-expressible). Skipped rows have promise_answer == answer and empty bindings.

Usage

python
import json
from datasets import load_dataset

ds = load_dataset("duoduoyeah/gsm8k-promise", "main")
row = ds["train"][0]
bindings = json.loads(row["promise_bindings"])   # promise_bindings is a JSON string

License & attribution

Derived from openai/gsm8k (MIT). Original GSM8K: Cobbe et al., 2021.