PavelGolikov/arbigraph

ArbiGraph

ArbiGraph is a benchmark generator for evaluating context management in language models and agents. It automatically builds verifiable directed task graphs whose nodes are math, Python tracing, or GSM-style tasks, and whose edges pass one task's output into another task's input.

The datasets uploaded here are example benchmark datasets generated with ArbiGraph. They are meant both for direct evaluation and as concrete examples of what the generator can produce. The generator source code and instructions for creating additional datasets are available in the GitHub repository: https://github.com/pavelgolikov/ArbiGraph.

This Hugging Face repository groups 21 generated example datasets as 21 separate dataset configs. Each config is one task category and one graph topology. There is no combined default dataset because the category/topology pairs are intended to be evaluated separately.

Loading

Load a specific category/topology dataset by config name:

from datasets import load_dataset

dataset = load_dataset("PavelGolikov/arbigraph", "math_six_chain_target")

All configs use the test split.

Intended Use

These datasets are designed to evaluate context management in an agent or evaluation harness. They are not primarily intended to test whether a model can solve math, Python tracing, or GSM-style tasks in isolation.

The intended setup is an agent with access to a calculator tool, not a Python interpreter. If the agent can run arbitrary Python, many Python tracing tasks can be executed directly, which changes the benchmark from context management into tool execution.

The tasks are deliberately easy to verify:

• —Math tasks are directly computable using SymPy or pure Python reference implementations.
• —Python tracing tasks are directly executable.
• —GSM-style tasks have deterministic generated answers.

This is intentional. ArbiGraph uses naturally parameterized, easily verifiable tasks so that task outputs can be chained through arbitrary graph topologies. The main difficulty is tracking relevant context, propagating intermediate outputs, and producing the requested target answer.

If your agent solves these datasets reliably, you can use the ArbiGraph generator to create harder settings, including larger graphs, longer lists, larger numeric magnitudes, or new topology files.

ArbiGraph can also be used as a tool-use compliance check. For example, one can give an agent access to a Python interpreter while instructing it to use only a calculator, then inspect whether it follows that restriction. This should be treated as an instruction-following or harness-compliance test, not as a general measure of trustworthiness.

Task Sources

The benchmark data is generated by ArbiGraph from deterministic task generators. The task pools use the following sources and reference implementations:

• —Math tasks are ArbiGraph tasks with SymPy or pure Python reference implementations.
• —Python tracing tasks are adapted from LeetCode solution snippets, selected from newfacade/LeetCodeDataset and tracked by LeetCode problem id.
• —GSM-style tasks are adapted from GSM-Symbolic templates.

The current HF datasets contain generated ArbiGraph prompts and generated answers. Please also check the upstream licenses and terms for the task sources when reusing or redistributing derived versions:

• —SymPy: https://www.sympy.org/
• —LeetCode solution dataset: https://huggingface.co/datasets/newfacade/LeetCodeDataset
• —GSM-Symbolic: https://github.com/apple/ml-gsm-symbolic

Configs

Schema

Each row has the following fields:

• —id: stable row id in the form category/topology/target_native_task_id/sample_idx.
• —category: one of math, python, or gsm.
• —topology: graph topology name.
• —target_node_id: target graph node id.
• —target_native_task_id: native task id inside the target category.
• —target_native_task_name: native task name inside the target category.
• —sample_idx: sample index for that target native task.
• —prompt: full benchmark prompt.
• —target_output_json: JSON-encoded target output value.
• —node_outputs_json: JSON-encoded list of all node outputs.

Parse the ground truth fields with json.loads:

import json

target_output = json.loads(dataset["test"][0]["target_output_json"])
node_outputs = json.loads(dataset["test"][0]["node_outputs_json"])

Topologies

The topology files used to generate these datasets are included under topologies/<category>/.

Each category/topology pair has:

• —topologies/<category>/<topology>.json: machine-readable graph definition.
• —topologies/<category>/<topology>.svg: rendered graph diagram.

The seven topology names are:

• —single_target_baseline
• —three_independent_target
• —three_chain_target
• —six_independent_target
• —six_chain_target
• —four_way_fan_in_target
• —two_branch_recombine_target

Version Note

The accompanying paper used an earlier ArbiGraph version with an older \boxed{...} output format and different graph topologies. The paper-era snapshot is preserved in the GitHub repository under the tag paper-v1-boxed.

Current datasets use JSON-style task outputs:

task_j_out = {"result": <result here>}

Links

• —GitHub: https://github.com/pavelgolikov/ArbiGraph
• —Paper: https://arxiv.org/abs/2607.20764
• —Paper-era raw result JSON archive: https://drive.google.com/drive/folders/10ix3KcGRF02N3I1C1QGSYWwTV4n2QOLy?usp=sharing

Citation

@misc{golikov2026arbigraph,
      title={ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management},
      author={Pavel Golikov and Evgenii Opryshko and Gennady Pekhimenko and Mark C. Jeffrey},
      year={2026},
      eprint={2607.20764},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.20764},
}