rvashurin/coqar-clarifications

CoQAR Clarifications

This dataset pairs 1,000 CoQAR development questions with their original stories and stories damaged by sentence deletion. Each of the resulting 2,000 inputs has five sampled model clarifications. Two configurations reuse the same generated additions and differ only in where those additions are placed.

Each configuration contains 1,000 damaged and 1,000 undamaged rows. The two configurations contain the same 10,000 sampled additions, rendered in different positions; they are not separate generation runs.

from datasets import load_dataset

appended = load_dataset("rvashurin/coqar-clarifications", "appended", split="dev")
inserted = load_dataset("rvashurin/coqar-clarifications", "inserted", split="dev")

Construction

The source is the CoQAR development JSON from Orange-OpenSource/COQAR, pinned to revision 1f21713f6d3152232bb1921a3767d2b7fa7255b4:

• —Source file: data/CoQAR/dev/coqar-dev-v1.0.json.
• —Source SHA-256: 3d093e1bf45e44f7cc4e3b8e07a37ded9162ded4ac88e90bb9e4c5657be3251f.
• —Subset selection seed: 20260911.
• —Selection: 1,000 distinct eligible question turns, covering 437 stories, after excluding the ten turns used in the earlier pilot. This is a turn-level exclusion; other turns from the same stories may be present.
• —Question: the first nonempty CoQAR out-of-context rewrite for each selected turn.
• —Damage: remove every complete sentence that overlaps the original annotated answer rationale. Sentence spans and rationale spans use the original story's character coordinates. Surviving characters, including whitespace between deleted sentences, are preserved.

The row flag is_underspecified is exactly condition == "damaged". It records nominal underspecification induced by the construction, not a judgment that the question is unanswerable. The damaged story can still contain enough information to answer the question. Likewise, undamaged does not guarantee that every rewritten question is well specified or correctly answerable.

Clarification generation

The two-step v2 pipeline used gpt-5.6-luna with reasoning effort low and temperature 1. First, the model produced a menu of possible interpretations, including an already-answerable state when appropriate. Five interpretation draws were then made with replacement from the normalized model-provided weights, using reproducible draws derived from seed 20260911. Each selected interpretation was passed to a separate clarification-generation call. Duplicates and empty additions were retained.

The model saw the input story and the question; the rendering step also received the interpretation menu and selected interpretation. It did not receive the original gold answer, deleted sentences, or rationale offsets. Those offsets are used later to construct the inserted configuration.

Five draws share the first-stage menu. In particular, an already-answerable menu can produce five empty additions; these are not five independent answerability judgments. The menu weights and generated additions are model outputs, not calibrated estimates of a human clarification distribution.

Four structural failures were recovered with their initial failures and recovery histories preserved. Outputs were not filtered or resampled for semantic quality, novelty, or diversity. Two menu errors involving reserved interpretation IDs caused three damaged-input empty additions; the affected records remain present with contract errors.

Placement rules

In appended, a nonempty trimmed addition is applied as input_story + "\n\n" + applied_addition. An empty addition leaves the input story exactly unchanged. This reproduces the original generation run's story construction.

In inserted, the same trimmed addition is inserted:

• —For a damaged input, at the start of the first removed sentence, expressed in the damaged input's coordinates.
• —For an undamaged input, immediately after the last original sentence that overlaps the rationale span. There is no random placement.

A nonempty addition is separated from each nonempty side of the input by a paragraph separator ("\n\n"). Every existing input character is preserved, including whitespace at the insertion boundary. An empty addition leaves the input exactly unchanged. Placement moves the addition; it does not regenerate its wording to fit the new location.

Fields

Each row is one context–question input and contains the following fields:

Each element of clarifications contains:

Character offsets are zero-based; span ends are exclusive. They index Unicode characters, not encoded bytes. Original rationale and sentence coordinates refer to original_story; clarification and inserted-block coordinates refer to clarified_story. Request and response fields describe the original generation calls, including when those additions are later placed in inserted stories.

Assessment and limitations

Damaged outputs were assessed by Luna with reasoning effort none, using the appended stories. These assessments are automated judgments, not human validation. Undamaged outputs were not audited by that step. The inserted stories have not been re-audited; any retained audit pertains to the appended version, as indicated by audit_applies_to_placement.

There are 4,550 unchanged undamaged samples and 1,608 unchanged damaged samples in each configuration. In the damaged condition, 321 inputs contribute five unchanged samples each; the remaining three unchanged samples are the recorded menu-ID errors. Empty additions are preserved in both placements.

The original gold answer is not a target label for a hypothetical clarified story: a sampled addition may establish a different answer. Model additions may be incorrect, contradictory, or unnecessary. Moving them into a passage can also affect coherence. The damage flag, model menu, audits, and source gold answer represent different kinds of information and should not be treated as interchangeable ground truth.

Original stories, rationale annotations, and original answers are exposed for analysis. An evaluation intended to measure behavior on damaged inputs should keep those reference fields out of model inputs.

Reproduction and provenance

The bundled `code/export_hub.py` constructs both configurations from the original saved experiment artifacts without further model calls. `code/publish_hub.py` publishes the prepared release. The original large validation JSONs are local run artifacts, identified by hashes in provenance.json; they are not included in this Hub repository. The published rows retain the input text, raw additions, exact requests/responses and insertion offsets needed to reconstruct either placement directly. The bundled processing sources and prompts document seeded selection, sentence removal, interpretation sampling, API processing, structural recovery, and export. Source and artifact hashes trace the published records back to the run. Public provenance omits local absolute paths and host identity.

Licensing and attribution

The CoQAR authors' license description assigns CC BY-SA 4.0 to their annotations and Apache 2.0 to their code. The original CoQA passages retain their domain-specific terms: CC BY-SA 4.0 for literature/Wikipedia, MSR-LA for MCTest, RACE-specific terms for examination passages, and Apache terms for CNN material. See the official CoQA license description and this repository's LICENSE.

This release is a derivative research dataset. Its sentence deletions, model-generated additions, and alternative placements are changes to the source material. Publishing these additions does not relicense the underlying passages. The license metadata is other to reflect the inherited mixture of terms; it is not a declaration that all components share one permissive license.

Citations

Please cite the source datasets:

@inproceedings{brabant-etal-2022-coqar,
  title = "{C}o{QAR}: Question Rewriting on {C}o{QA}",
  author = "Brabant, Quentin and Lecorv{\'e}, Gw{\'e}nol{\'e} and Rojas Barahona, Lina M.",
  booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
  year = "2022",
  pages = "119--126",
  publisher = "European Language Resources Association",
  url = "https://aclanthology.org/2022.lrec-1.13/"
}

@article{reddy-etal-2019-coqa,
  title = "{C}o{QA}: A Conversational Question Answering Challenge",
  author = "Reddy, Siva and Chen, Danqi and Manning, Christopher D.",
  journal = "Transactions of the Association for Computational Linguistics",
  volume = "7",
  year = "2019",
  pages = "249--266",
  doi = "10.1162/tacl_a_00266",
  url = "https://aclanthology.org/Q19-1016/"
}