mattwang123/chrononauts-retrieval-sft-202407-500

Chrononauts Retrieval-Grounded SFT (202407, n=500)

500 Think-SFT examples that teach a language model to ground its answers in retrieved documents rather than relying on parametric knowledge. Schema is byte-identical to `allenai/Dolci-Think-SFT-7B` so it can be folded into existing Think-SFT training pipelines without modification.

Proof-of-concept dataset for the Chrononauts project, targeting the 202407 model cutoff.

Schema

Three top-level fields per example:

{
  "messages": [
    {"role": "user", "content": "Here is some context:\n[1] <title>: <doc text>\n[2] ...\n\nNow answer this question using the context above. ...\n\nQuestion: ..."},
    {"role": "assistant", "content": "<think>\n...reasoning that cites docs by [N]...\n</think>\n\n<final answer>"}
  ],
  "dataset_source": "hotpot_qa:distractor | rajpurkar/squad | framolfese/2WikiMultihopQA",
  "id": "<source>_<source-native-id>"
}

messages[1].content follows the Olmo Dolci-Think-SFT convention: a <think>...</think> block with grounded reasoning (citing docs by bracket number), then the final answer in plain prose. No <answer> tags, no **Answer**: marker.

Source mix

All three sources are Wikipedia-grounded and built from Wikipedia dumps that predate the 202407 cutoff by years (HotpotQA: Oct 2017 dump; SQuAD: pre-2018; 2Wiki: 2020). No per-passage temporal filter was applied — provenance is safe by construction for this cutoff. For tighter cutoffs (e.g. 201305) a per-passage filter would be required.

Generation pipeline

1. 1.Pool building — sampled examples from the three sources (1.3× oversample = 650 candidates).
2. 2.Teacher generation — gpt-5-2025-08-07 (OpenAI), prompted to produce <think>...</think>\n\n<answer> strictly grounded in the provided docs, with explicit bracket citations.
3. 3.Quality filter — kept only examples where the post-</think> answer string-matches the gold answer after normalization (lowercase, strip punctuation, strip leading articles, containment check). 89.2% overall pass rate (HotpotQA 90.0%, SQuAD 89.2%, 2Wiki 86.9%).
4. 4.Assembly — downsampled to exactly 300/100/100 per source, written in Dolci-Think-SFT-7B schema.

Per-example provenance (gold answer, source id, teacher model, cutoff) is in sft_500.jsonl.manifest.jsonl, keyed by id.

Usage

from datasets import load_dataset

ds = load_dataset("mattwang123/chrononauts-retrieval-sft-202407-500", split="train")
print(ds[0]["messages"][1]["content"])

The schema matches allenai/Dolci-Think-SFT-7B, so any trainer that consumes that dataset (e.g. open-instruct, axolotl with the Think format) consumes this one identically.

Intended use

Mixing into Think-SFT training stages on Chrononauts-3B (or any base/instruct model) to add explicit retrieval-grounding behavior. The student model learns to:

• —Read numbered context documents
• —Cite them by [N] during <think> reasoning
• —Produce a final answer that's traceable back to the docs

Limitations

• —Only ~500 examples — proof-of-concept scale. Useful for verifying the recipe; for production training, scale up by re-running the pipeline with a larger TARGET_TOTAL.
• —No abstention training — an earlier design included SQuAD v2 unanswerables to teach "say I don't know when the doc doesn't contain the answer," but it was dropped. The current dataset assumes every question is answerable from the docs.
• —No per-passage temporal filter — safe for the 202407 cutoff by source-provenance, but cannot be reused as-is for tighter cutoffs without additional filtering.
• —Teacher knowledge leakage — gpt-5-2025-08-07 has knowledge through ~mid-2025. The teacher system prompt forbids using outside knowledge, but subtle leakage in reasoning is possible. The string-match filter catches blatant cases where the answer disagrees with the gold.

Citation

If you use this dataset, please cite the upstream sources:

@inproceedings{yang2018hotpotqa,
  title={HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering},
  author={Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D.},
  booktitle={EMNLP},
  year={2018}
}

@inproceedings{rajpurkar2016squad,
  title={SQuAD: 100,000+ Questions for Machine Comprehension of Text},
  author={Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy},
  booktitle={EMNLP},
  year={2016}
}

@inproceedings{ho2020constructing,
  title={Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps},
  author={Ho, Xanh and Duong Nguyen, Anh-Khoa and Sugawara, Saku and Aizawa, Akiko},
  booktitle={COLING},
  year={2020}
}

Schema convention adopted from `allenai/Dolci-Think-SFT-7B`. Teacher: OpenAI gpt-5-2025-08-07.