Emulated-Inc/long-context-retrieval-training-pool

Long context retrieval training pool

Long prompts with short, checkable answers. Each row is one complete message: a task instruction, a long body of text that hides what the question is about, and the question itself, together with every string an answer has to contain for it to be right. The bodies run from four thousand to thirty-two thousand tokens. Three sources, laid out twice. Train on either layer or on both.

pool.jsonl

Every source rewritten into one shape, 20000 rows, one JSON object per line, with these fields. The file is written one source after another, the generated rows first, then the generated question answering, then the collected questions, so a sequential read of its first rows trains on one source.

Checking an answer needs no model. Six of the seven tasks are scored by how many of the strings in answer appear in the output, compared case-insensitively as substrings, and the seventh, question answering, is right if any one of them appears, and every question-answering row of this pool carries exactly one string, so the two rules coincide here. That is the whole rule, so a row is usable as a verifier reward as it stands.

tokens and length are counted with one particular subword tokenizer, the one whose tokenizer.json has the sha256 12bac982b793c44b03d52a250a9f0d0b666813da566b910c24a6da0695fd11e6. A different tokenizer will count a few percent differently, which is why chars is there as well.

The seven kinds of question

How the rows fall across the tasks and the lengths:

sources/

The same data in the shape its own producer writes, one directory per source. The two generated sources are written as the generator writes them, the message in items.jsonl and the answer in answers.jsonl keyed by the same id, so a set can be handed to a model without the strings it is scored against. The downloaded source holds the files at the paths, in the parquet format and with the columns its own repository publishes, rewritten with pyarrow so that a removed row can be removed at all. Use this layer if you want a field the rewritten one drops, such as the supporting facts and the question type of the downloaded set.

The sources

sources/generated

Questions of the six synthetic kinds, generated for this pool under seed 20260914 by a port of the published generators of Hsieh and colleagues (arXiv 2404.06654, github.com/NVIDIA/RULER, Apache 2.0), with their templates and their parameters unchanged. The bodies of the retrieval questions are sentences of public-domain books from sedthh/gutenberg_english at revision 28973b04f28fd7be4a6186a042bc26159d4366ca, MIT, and the needle keys and the listed words are ordinary words of those books. 15000 rows. Provenance class rule-generated, licence cc-by-4.0.

Worth knowing. One generator drew all of it, so it is one distribution rather than a survey of long-context question styles, and the five lengths are equally weighted rather than sampled.

sources/generated_qa

Question answering over distractor documents, generated for this pool under the same seed by the same port, over the training split of rajpurkar/squad at revision 7b6d24c440a36b6815f21b70d25016731768db1f, CC BY-SA 4.0. The document a question is answered from is hidden among documents of other articles. 2500 rows. Provenance class rule-generated, licence cc-by-sa-4.0, since the documents are the dataset's own paragraphs.

Worth knowing. The answers are the aliases the original dataset publishes, and an alias that did not survive the whitespace normalisation of its own paragraph was dropped, so every string in answer really occurs in the document it comes from.

sources/hotpot_qa

Multi-hop question answering over ten documents, where two of them carry the answer and eight are distractors, from hotpotqa/hotpot_qa at revision 1908d6afbbead072334abe2965f91bd2709910ab, configuration distractor, training split, files distractor/train-00000-of-00002.parquet, distractor/train-00001-of-00002.parquet, CC BY-SA 4.0. 90447 of its questions are in the raw layer. 68718 of them have an answer that is written in one of their own documents, which is what this pool's substring rule needs, and 2500 of those are in the curated layer, rendered into the same long shape as the generated rows: their own ten documents plus distractors drawn from the 283771 documents of the whole split, under the same question-answering template. Provenance class collected, licence cc-by-sa-4.0.

Worth knowing. This is the one source written by people: the questions were authored by crowd workers over Wikipedia and the answers are theirs. 18805 of its questions are left out of the curated layer because their answer is not found by the substring rule in any of their own documents once the rule has dropped every document that holds a character outside plain ASCII: a yes or no answer is seldom written down as such (the 90 question-answering rows of the curated layer whose answer is yes or no pass because the word occurs in their documents), and an answer that sits only in a dropped document counts here too, which is by far the larger share. A further 2924 carry such a character in the answer itself or in every one of their documents. Both groups are in the raw layer only. The curated layer's documents are plain ASCII for that reason, while a few of its questions carry an accented letter. Counted over the shipped files: 2671 of the 18805 answer yes or no, 16134 have their answer in a dropped document, and 76 curated questions carry an accented letter, 73 from this source and 3 from the generated question answering.

Provenance and licences

Two of the three sources were produced by a program that draws a body of text and hides a fact in it, which is the provenance class rule-generated, and they are most of the rows. The third is classed collected because its questions and answers were written by people. No row anywhere in the pool was written by a language model.

The pool as a whole is offered under cc-by-sa-4.0, the term its sources compose to. The generated synthetic rows are cc-by-4.0 and carry no share-alike obligation of their own, and the two question-answering sources are cc-by-sa-4.0 because their documents are the original datasets'. Every rewritten row carries its own term in the licence field and each directory under sources/ is one source, so a subset under a single licence can be selected. Attribution goes to Hsieh and colleagues for the generators, to the maintainers of sedthh/gutenberg_english for the book text, to Rajpurkar and colleagues for rajpurkar/squad and to Yang and colleagues for hotpotqa/hotpot_qa.

Filtering

Nothing was removed. Every row was compared against a held-out list that is not distributed here, three ways: the whole message, the short labels in identity that say what the row hides, and each document inside a question-answering row. None of the three matched, which is what the construction expects, since the books, the words, the coded tokens and the articles this pool is built from were split away from that list before any row existed. No task, no length and no difficulty was selected for or against.