Emulated-Inc/logical-reasoning-training-pool

Logical reasoning training pool

Public logical-reasoning problems with checkable answers, from six datasets, read at the pinned revisions named below and laid out twice. Train on either layer or on both.

pool.jsonl

Every source rewritten into one shape, 579910 rows, one JSON object per line. A row is one of three kinds and carries the fields its kind needs.

By kind: 565851 entailment rows, 7859 choice rows and 6200 open rows. Rows are deduplicated across sources on the problem text together with its option set, keeping the first source that carries the problem in the order of the sections below. The file is written one source after another in that same order, so a sequential read of its first rows trains on one source.

Three of the sources publish their own label words. They are written into the common vocabulary here: unknown and neutral become Uncertain, entailment becomes True, and contradiction and not entailment become False. The raw layer keeps whatever each source wrote. One source reasons under the closed-world assumption, so its False means the problem's rules do not entail the conclusion rather than that they entail its negation, which its section below repeats.

sources/

The same data as published, minus only the rows the filtering below removed, 1096561 rows, one directory per source, holding the files at the paths, in the format and with the columns its own repository publishes. Nothing else here was renamed, reshaped, relabelled or deduplicated. Use this layer if you want a field the rewritten one drops, such as the proof trees, the first-order- logic formalisations or the worked chains of reasoning, or if you would rather map the labels yourself.

The sources

sources/folio

Expert-written first-order-logic entailment problems, each a block of premises in plain English, one conclusion, and a label saying whether the premises make the conclusion true, false or neither, with a parallel formalisation of both in first-order logic. From tasksource/folio at revision 295b95fb4fe9be4ff3f933b73142d142cf6b2c97, files folio_v2_train.jsonl. 219 rows here, of which 219 also appear in pool.jsonl. Provenance class human, licence cc-by-sa-4.0. Its own fields are storyid, premises, premises-FOL, conclusion, conclusion-FOL, label and exampleid.

Worth knowing. The project repository licenses the work under Creative Commons Attribution- ShareAlike 4.0 while the authors' own dataset page tags it mit, and the stricter of the two is what the pool composes against. Several rows share one block of premises and differ only in the conclusion, which is how the set is built, so a random split of these rows is not an independent split.

sources/logiqa2

Logical reasoning questions from the Chinese civil service entrance examination, professionally translated into English and checked by human experts, each a short passage, a question about it and four options. From datatune/LogiQA2.0 at revision edbf6440d36a03f715c0c9d479f421f2ece87d6a, files MRC/train.txt. 8438 rows here, of which 7859 also appear in pool.jsonl. Provenance class human, licence unknown. Its own fields are id, text for the passage, question, options, answer as a zero-based index, type.

Worth knowing. The copy read here tags itself mit and the project repository publishes no licence file at all, so the tag covers the packaging rather than the examining bodies' rights in the underlying questions. The published train file, 12567 rows upstream before the filtering below, repeats 903 of its problems, the same passage, question and options behind a different id and never a byte-identical line. Among the 8438 rows kept here the rewritten layer drops 570 such repeats by its normalised, order-insensitive key and the raw layer keeps them. The file also reuses 148 of its ids, each on two different problems, so id is not a key.

sources/proofwriter

Generated theories of facts and rules in plain English with a question about them, under the open- world assumption, so the answer is true, false or unknown, at reasoning depths from zero to five plus a hand-written natural-language variant. From tasksource/proofwriter at revision ee4d22d49b8486786cbc01c0b6f6f6266e56f3bd, files data/train-00000-of-00002-6176cf3d78f48858.parquet, data/train-00001-of-00002-901eaba2151bde61.parquet. 585552 rows here, of which 376806 also appear in pool.jsonl. Provenance class rule-generated, licence unknown. Its own fields are id, maxD, NFact, NRule, theory, question, answer, QDep, QLen, allProofs for the proof trees, and config naming the depth.

Worth knowing. The release ships no licence file and the copy read here carries no licence tag. Its own README describes it as an updated version of the RuleTaker datasets, which the Allen Institute for AI publishes under Apache 2.0, so redistribution rests on that statement rather than on a licence for this release. Many rows share one theory and differ only in the question.

sources/ruletaker

The earlier generated theories the set above updates, under the closed-world assumption, so a question is either entailed by the theory or not, at reasoning depths from zero to five plus a hand-written natural-language variant. From tasksource/ruletaker at revision a3e0880baeb6ec3d478f4c4d85afe04b21b6cf7f, files data/train-00000-of-00001-52adaa842dd7ed92.parquet. 480152 rows here, of which 176827 also appear in pool.jsonl. Provenance class rule-generated, licence apache-2.0. Its own fields are context, question, label of entailment or not entailment, and config.

Worth knowing. Closed world, so its not entailment label covers both a false statement and one the theory says nothing about, which the two labels of the rewritten layer keep apart from the three- way labels of the other sources. It overlaps the set above by construction and the rewritten layer deduplicates across the two.

sources/logicnli

Generated first-order-logic entailment problems in plain English, built so that the answer follows from the logic rather than from world knowledge, with facts about named people followed by rules over them. From tasksource/LogicNLI at revision ac9d30ac7351c5f9bb31970b7ae82182b8b4f5e8, files data/train-00000-of-00001-05a75d620743c1d1.parquet. 16000 rows here, of which 11999 also appear in pool.jsonl. Provenance class rule-generated, licence unknown. Its own fields are premise, hypothesis, and label of entailment, contradiction, neutral or self_contradiction.

Worth knowing. Neither the copy read here nor the project repository declares a licence. A quarter of its rows carry the label self_contradiction, which says the premises contradict each other rather than saying anything about the hypothesis. Those rows have no place in a three-way label vocabulary and are dropped from the rewritten layer and kept in the raw one.

sources/knights_and_knaves

Generated knights-and-knaves puzzles, in which every inhabitant either always tells the truth or always lies and their statements about one another fix who is which, from two to eight inhabitants, each with the unique solution written out and a worked chain of reasoning beside it. From K-and-K/knights-and-knaves at revision 2f68547989981b1af37cb3dde5fdefa847aa8619, files train/people2_num200.jsonl, train/people3_num1000.jsonl, train/people4_num1000.jsonl, train/people5_num1000.jsonl, train/people6_num1000.jsonl, train/people7_num1000.jsonl, train/people8_num1000.jsonl. 6200 rows here, of which 6200 also appear in pool.jsonl. Provenance class rule-generated, licence cc-by-nc-sa-4.0. Its own fields are quiz, names, knightknave for the wording used, solution as a list of booleans, solutiontext, solutiontextformat, statements, index, and cothead, cotrepeatsteps and cotfoot for the worked reasoning.

Worth knowing. Non-commercial and share-alike, which is what makes the pool as a whole non- commercial. The puzzles come from one generator, so their surface wording repeats heavily across rows and a model can learn the wording rather than the reasoning.

Provenance and licences

Two of the six sources are problems written by people, one set by domain experts and one by the boards of a national examination and its professional translators. The other four are the output of programs that build a problem and its answer together, which is the provenance class rule- generated and which is most of the rows. Nothing in the pool was generated by a language model, and no answer anywhere in it was produced by a model: every key is either a human annotation or the generator's own record of the answer it built the problem around.

The pool as a whole is offered under cc-by-nc-sa-4.0, which is the most restrictive term its sources compose to. The sources themselves are cc-by-sa-4.0 for the expert-written entailment set, apache-2.0 for the closed-world generated set, cc-by-nc-sa-4.0 for the knights-and-knaves set, and an undeclared licence for the examination questions, the open-world generated set and the generated entailment set, none of which publishes a licence at its own project repository. Each rewritten row carries its own in the licence field and each directory under sources/ is one source, so a subset under a single licence can be selected. Attribution for the share-alike sources goes to the authors named in each source's own citation, and questions taken from national examinations remain the property of the boards that wrote them.

Filtering

Rows whose text duplicated or overlapped the text of a problem in a held-out collection were removed before publication, from both layers alike, by an exact whole-text match (1257 rows) followed by a word 8-gram overlap check (3654 rows). Both checks read the premises and the conclusion, or the passage and the question, and never the options. An 8-gram carried by more than one held-out problem is a stock phrase rather than a problem, so it was left out of that check, and the 4286 rows whose only overlap was such a phrase were counted and kept. The held-out collection is not distributed here. Nothing else was filtered: no reasoning depth, no subject and no difficulty was selected for or against, so the pool still holds the easiest depth-zero generated rows beside the hardest examination questions, and the subject field is how to select what you want. That field is the source's own configuration label. For the ProofWriter rows the labels are depth-0, depth-1, depth-2, depth-5, NatLang and depth-3ext-NatLang, and the last of these is a mixed bag holding the depth-3 theories together with some depth-1 and depth-2 ones (their ids carry D3, D2 and D1), so a ProofWriter depth-3 subset cannot be selected by subject alone, while the RuleTaker rows carry depth-3 and depth-3ext labels of their own.

Two warnings a reader should carry into training. Four of the six sources come from generators, so their wording repeats heavily and a model can learn the surface pattern rather than the reasoning, which is the finding the knights-and-knaves authors published about their own set. And several sources build many rows on one block of premises, differing only in what is asked, so a random split of these rows does not give independent train and held-out halves.