nutshells3/livegraph-matched-benchmark

1---2license: apache-2.03task_categories:4  - text-generation5language:6  - en7tags:8  - reasoning9  - graph-reasoning10  - synthetic11  - verifier12  - paired-evaluation13pretty_name: LiveGraph Matched Graph-vs-Linear Benchmark14---15 16# LiveGraph Matched Graph-vs-Linear Benchmark17 18This release contains the frozen synthetic datasets used by *LiveGraph: Typed19Graphs versus Linear Traces*. It supports a20controlled comparison between a typed-graph rendering and a chronological21linear-trace rendering of the same Boolean relational-propagation records.22 23## Main matched benchmark24 25- `data/semantic_train.jsonl`: 480 semantic training records, depths 1--10.26- `data/semantic_heldout.jsonl`: 3,200 clean held-out records, depths 1--32.27- `data/semantic_eval_variants.jsonl`: 19,200 semantic evaluation variants.28- `data/sft_{typed_graph,linear_trace}.jsonl`: paired SFT rows.29- `data/eval_{typed_graph,linear_trace}.jsonl`: paired evaluation rows.30- `manifest.json`: frozen counts, construction invariants, and file hashes.31 32The variants are `clean`, `edge_order`, `node_rename`,33`extra_distractors`, `one_path_edge_flip`, and `edge_delete`. The generator34balances labels within depth and holds the semantic record and full-chat token35count fixed across the two registered renderings. Ordinary visible trailing36`PAD` markers are part of the graph rendering package; this is not a pure37graph-topology intervention.38 39## Additional frozen subsets40 41- `data/grpo_v1/`: exact-verifier GRPO train, ID, and OOD sets.42- `data/native_robustness_v1/`: native-thinking robustness subset.43- `data/edge_delete_unknown_v1/`: deletion-only UNKNOWN evaluation.44- `data/subsets_v3/`: clean and diagnostic subsets.45- `data/relassum_v3/`: auxiliary relation/assumption evaluation artifacts.46 47## Integrity48 49Verify the frozen main files from this directory:50 51```bash52sha256sum -c SHA256SUMS53```54 55For the original main files, the frozen hashes are also recorded in56`manifest.json` and `SHA256SUMS`. Derived subset directories carry their own57manifests.58 59The byte-frozen top-level manifest retains two original absolute tokenizer and60chat-template provenance strings so that its paper-reported SHA-256 remains61verifiable. They are inert local paths; use the public base model files rather62than those locations.63 64## Scope and limitations65 66The data are synthetic. `supports` preserves a Boolean value and `refutes`67negates it along one queried path. These tasks do not establish real-world68reasoning quality, theorem validity, source-claim truth, or end-to-end OAE69assurance. Auxiliary relation/assumption labels are not human gold.70 71## License and citation72 73The released dataset is provided under Apache License 2.0. Cite the LiveGraph74paper and this dataset release. A final arXiv identifier can be added to this75card after submission.76