hoololi/llm-agent-harness-reliability-next-prime

LLM Next Prime Harness Dataset

This dataset contains raw observations from an experiment studying how the agent harness affects reliability when an LLM has access to a deterministic tool.

The task is deliberately simple and objectively verifiable:

What is the smallest prime number that is strictly greater than n?

The deterministic tool computes the correct answer with a local Python next_prime(n) function. The experiment asks whether failures come from the model, the provider response, the tool boundary, or the harness logic around the tool.

Dataset composition

The published Parquet file combines two independent runs:

2 runs × 500 input cases × 6 models × 4 harnesses × 1 repetition = 24,000 rows

Source JSONL files:

• —llm_harness_next_prime_20260825_500cases.jsonl
• —llm_harness_next_prime_20260827_500cases.jsonl

The Parquet file adds a source_file column identifying the originating JSONL file.

Inputs

The 500 inputs are deterministic and cover five contiguous windows:

• —1,000–1,099
• —10,000–10,099
• —100,000–100,099
• —1,000,000–1,000,099
• —100,000,000–100,000,099

The same inputs are used for every model and harness.

Models

The experiment includes six small or inexpensive models available through OpenRouter:

• —nvidia/nemotron-3.5-lightning
• —poolside/laguna-xs-2.1
• —inception/mercury-2
• —thinkingmachines/inkling-small
• —upstage/solar-pro4
• —google/gemini-3.7-flash

Harnesses

The methodological principle is:

H(n+1) = H(n) + one explicit harness mechanism.

H0 — Minimal tool loop

The model receives the question and one available tool, next_prime. If a tool call is emitted, the harness parses the JSON arguments, executes the tool, sends the result back, and records the model's final response. No repair or validation beyond basic execution is applied.

H1 — Validated tool input

H1 is identical to H0 except that tool arguments are strictly validated before local tool execution. For this task, the accepted argument object is exactly {"n": int}. Strings, booleans, floats, missing fields, and extra fields are rejected rather than coerced.

H2 — Validated final output

H2 is identical to H1 except that the final model response is validated. The final answer must be non-empty, contain an extractable integer, and, when a successful tool output exists, the extracted final answer must match the last successful tool output. This validation does not use the ground-truth label.

H3 — Controlled recovery

H3 is identical to H2 except for one deterministic recovery rule: if final-answer validation fails and a successful tool output already exists, the harness uses the last successful tool output as the recovered final answer. H3 does not retry the model and does not use ground truth for recovery.

Important fields

Core fields:

• —experiment_id
• —run_id
• —source_file
• —run_order_index
• —task_type
• —case_id
• —input_number
• —correct_next_prime
• —magnitude_group
• —model, provider, model_family, model_label
• —harness, harness_description
• —repetition_id
• —question

Answer and evaluation fields:

• —raw_response: final response text returned by the model, if any.
• —extracted_answer: answer extracted from raw_response only.
• —final_answer: value used for final correctness evaluation.
• —recovered_answer: deterministic H3 recovery value, if recovery was applied.
• —final_answer_source: raw_response, recovered_tool_output, or none.
• —final_result_correct: whether final_answer equals the deterministic ground truth.
• —extracted_answer_correct: whether extracted_answer equals the deterministic ground truth.
• —tool_result_correct: whether the last successful tool output equals the deterministic ground truth.

Tool and harness fields:

• —tool_called
• —tool_call_count
• —tool_calls
• —tool_input
• —tool_output
• —tool_error
• —successful_tool_output_available
• —last_successful_tool_output
• —final_validation_success
• —final_validation_error
• —recovery_applied
• —recovery_reason
• —provider_response_error
• —provider_response_error_stage

Operational fields:

• —run_success
• —run_error
• —latency_seconds
• —input_tokens, output_tokens, total_tokens
• —cost
• —timestamp

Semantics of run_success

run_success and final_result_correct intentionally measure different things.

• —run_success describes whether the harness execution completed according to that harness' own rules.
• —final_result_correct describes whether the final answer value recorded for the row equals the deterministic ground truth.

For example, H0 has no final-answer validation. A model can return a malformed or incorrect final answer while the run is still technically successful. Conversely, H2 treats final-answer validation failure as a harness execution failure. H3 may recover deterministically from an already successful tool output while preserving the raw failed final response and validation error.

Parquet notes

The JSONL files preserve the raw record structure. In the Parquet file, nested tool_calls traces are serialized as JSON text for portability. Exact textual representations of potentially large model-produced numeric outputs are preserved in _raw columns such as final_answer_raw, extracted_answer_raw, and tool_output_raw.

Redaction

Provider error messages were redacted before publication to remove account-specific identifiers such as OpenRouter user IDs. Error codes, provider names, rate-limit messages, and endpoint/provider failures are otherwise preserved when available because they are useful for analysis.

Suggested analyses

This dataset can be used to study:

• —harness-level accuracy and run success;
• —differences between final-answer correctness and tool-output correctness;
• —controlled recovery effects in H3;
• —provider/infrastructure failures;
• —model-specific failure signatures;
• —stability across the two independent runs;
• —latency, tokens, and cost by model and harness.

Limitations

• —The experiment covers one deterministic task and one deterministic tool.
• —Results depend on the OpenRouter gateway, provider routing, model versions, and run dates.
• —The dataset is not intended as a definitive benchmark leaderboard.
• —H3 recovery is intentionally narrow and deterministic; it is not an open-ended retry policy.

License

Released under CC BY 4.0.