hoololi/llm-calculations-legacy-v01

⚠️ ARCHIVED DATASET This dataset is archived and no longer maintained. For a more robust, updated, and methodologically improved version of this work, please use the primary dataset: hoololi/llm-calculations

Local Arithmetic LLM Experiments

This dataset contains factual observations from a small local experiment comparing how different large language models answer arithmetic questions in two configurations:

1. 1.LLM only: the model receives an arithmetic question and answers without external tools.
2. 2.LLM + calculator: the same model receives the same arithmetic question, uses a local calculator-assisted flow, and returns a final answer.

The goal of this dataset is not to rank models or provide a subjective benchmark. It is intended as raw experimental data that others can analyze independently.

Motivation

Arithmetic is a useful setting for observing differences between direct model answers and answers produced with tool assistance. This dataset records both the raw model responses and extracted numeric results so that users can later compute their own metrics, such as accuracy, absolute error, latency trade-offs, or cost per correct answer.

No subjective labels are included.

Data collection setup

The experiment was run locally using OpenRouter as the LLM provider. The calculator tool was implemented locally and safely, without using Python eval.

For each arithmetic question and each model, the script recorded:

• —the manually recorded correct result;
• —the raw LLM-only answer;
• —the numeric result extracted from the LLM-only answer;
• —the raw calculator-assisted answer;
• —the numeric result extracted from the calculator-assisted answer;
• —latency for both modes;
• —token usage when available;
• —cost when available;
• —simple numeric differences between the two modes when computable.

The difference fields compare the two experimental modes with each other. They are not accuracy fields. Correctness should be evaluated against correct_result.

The input operations were stored in JSON format with two fields: operation and result. Only operation was sent to the models. The result value was computed locally before running the experiment, manually recorded in the operations file, exported as correct_result, and used as the source of truth for downstream analysis. correct_result was never included in the prompts sent to the models. No difficulty level, operation type, or pre-written expression was provided as input to the models.

Calculator-assisted mode

In calculator-assisted mode, the model was first asked to produce the arithmetic expression to calculate. The local calculator evaluated that expression, and the result was then sent back to the model so it could produce a final answer.

The calculator expression and calculator result are included in the dataset when available. These fields are factual observations from the tool-assisted flow, not pre-supplied answers.

Numeric extraction

For convenience, the dataset includes extracted numeric results from the raw model answers.

The extraction rule is simple:

• —take the last number appearing in the answer;
• —remove thousands separators such as commas;
• —convert to integer if possible, otherwise float;
• —return null if no number can be extracted.

The raw answer is always preserved, so users can apply their own extraction logic if desired.

File format

The dataset is stored as JSONL. Each line corresponds to one arithmetic question × one model.

Example record:

{
  "experiment_id": "arith_000001",
  "batch_id": "batch_2026_06_26_120000",
  "timestamp": "2026-06-26T12:00:00Z",
  "operation_id": "op_000001",
  "question": "What is 347 × 82?",
  "correct_result": 28454,
  "model": "example/model-name",
  "llm_only_answer": "347 × 82 = 28,454.",
  "llm_only_result": 28454,
  "llm_only_latency_seconds": 1.08,
  "llm_only_cost": null,
  "llm_only_prompt_tokens": 24,
  "llm_only_completion_tokens": 12,
  "llm_only_total_tokens": 36,
  "llm_only_error": null,
  "llm_with_calculator_answer": "347 × 82 = 28,454.",
  "llm_with_calculator_result": 28454,
  "llm_with_calculator_latency_seconds": 1.43,
  "llm_with_calculator_cost": null,
  "llm_with_calculator_prompt_tokens": 80,
  "llm_with_calculator_completion_tokens": 30,
  "llm_with_calculator_total_tokens": 110,
  "llm_with_calculator_error": null,
  "llm_with_calculator_calculator_expression": "347 * 82",
  "llm_with_calculator_calculator_result": 28454,
  "difference": 0,
  "latency_difference_seconds": -0.35,
  "cost_difference": null
}

Field notes

• —correct_result is the locally computed and manually recorded source-of-truth result from the operations file. It is included for analysis but was not provided to the models in the prompt.
• —difference = llm_only_result - llm_with_calculator_result
• —latency_difference_seconds = llm_only_latency_seconds - llm_with_calculator_latency_seconds
• —cost_difference = llm_only_cost - llm_with_calculator_cost
• —If either numeric result is unavailable, difference is null.
• —If either cost is unavailable, cost_difference is null.
• —difference = 0 means the two modes agreed; it does not by itself mean the answer is correct.
• —Error fields such as llm_only_error and llm_with_calculator_error contain technical errors when a model call or calculator-assisted step failed; otherwise they are null.
• —If a provider returns an empty response without an API error, the corresponding error field is set to empty_response in batches generated with the updated script.

What this dataset is not

This dataset does not include:

• —model rankings;
• —accuracy scores;
• —subjective ratings;
• —difficulty labels;
• —winner/better-model labels;
• —failure-type annotations.

It is intended to provide raw observations only.

Limitations

• —The dataset depends on the exact model versions and provider routing available through OpenRouter at collection time.
• —Latency can be affected by network conditions, provider load, OpenRouter routing, and rate limits.
• —Calculator-assisted mode involves more than one LLM call, so its latency and token usage are not directly comparable to a single-call LLM-only answer without taking that into account.
• —Numeric extraction is intentionally simple and may not match every possible model response format.
• —The correct_result field was computed locally before the run, manually recorded in the operations file, and should be treated as the source of truth for correctness analysis. It was not provided to the models in the prompt.

Intended uses

Possible analyses include:

• —comparing raw numeric outputs across modes;
• —measuring how often calculator assistance changes the final numeric answer;
• —estimating latency and cost trade-offs;
• —evaluating both modes against the correct_result field;
• —studying tool-use behavior through the recorded calculator expressions.

Reproducibility

The data was generated by a local Python script using OpenRouter and a local calculator implementation. The experiment does not require a web server, database, or remote deployment.