CoolFace
Datasetpublic

alxraun/ers-bench-nano

ERS-Bench-Nano A parallel QA dataset designed to evaluate the effects of ERS semantic compression by comparing answer accuracy between natural language source text and its lossy LLM-generated ERS artifact. Schema Field Type Description domain string Category: code, formal, science, humanities. sample_id string Sample identifier. source string Original natural language text. ers_artifact string ERS notation artifact. question_id string Query… See the full description on the dataset page: https://huggingface.co/datasets/alxraun/ers-bench-nano.

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes9downloads
Dataset Card

ERS-Bench-Nano

A parallel QA dataset designed to evaluate the effects of ERS semantic compression by comparing answer accuracy between natural language source text and its lossy LLM-generated ERS artifact.

Schema

FieldTypeDescription
domainstringCategory: code, formal, science, humanities.
sample_idstringSample identifier.
sourcestringOriginal natural language text.
ers_artifactstringERS notation artifact.
question_idstringQuery identifier.
question_typestringsimple (direct fact retrieval) or complex (multi-hop reasoning).
questionstringQuery text.
answerstringGround truth response.

Domains

  1. 1.Code: Structural and behavioral software contracts.
  2. 2.Formal: Regulatory norms and constraints.
  3. 3.Science: Causal links and reasoning chains.
  4. 4.Humanities: Ideological and logical structures.

Known Limitations & Empirical Observations

1. Methodological Facts

  • —ERS artifacts and QA pairs were generated independently, using only source.txt as input, without access to each other's output.
  • —ERS artifacts and QA pairs, including complex questions, were generated by the same frontier model.

2. Empirical Observations

  • —ERS artifacts in the dataset are the result of lossy generation: structural and causal relationships are preserved, while narrative elements and some low-level identifiers are dropped.
  • —For some simple questions that require direct fact retrieval, the ERS artifact may drop certain narrative elements and some low-level identifiers.
  • —For some complex questions that require multi-hop reasoning over the original source, the ERS artifact may already contain the explicit logical chain (e.g., A -> B => C), turning the task into direct fact retrieval.

3. Possible Interpretations

  • —Since ERS and QA were generated by the same model, its internal attention mechanisms may have highlighted the same structural nodes. This may have led some ERS artifacts to contain direct answers to questions that were designed to be complex.

Sources

Mapping and licensing for source materials are provided in THIRD-PARTY-NOTICES.md.