PranathReddy/solveall-literature-priors

SolveAll Literature-Grounded Priors

Dataset summary

SolveAll Literature-Grounded Priors is an English-language dataset of open-ended mathematical and scientific research problems paired with realistic user priors whose epistemic relationship to the literature is explicitly annotated. Each claim-bearing example is connected to one or more short passages from identified literature sources. The passages are used to classify the user's prior as contradicted, supported, qualified, or not settled by the available evidence. Neutral examples contain no asserted claim.

The dataset is designed to support research on whether an assistant responds to a researcher's framing in an evidence-sensitive way. In particular, it separates the truth status of a prior from the confidence with which the user presents it. This makes it possible to distinguish appropriate agreement from generic agreement, appropriate disagreement from generic skepticism, and calibrated uncertainty from unsupported certainty.

The current release contains the completed train split. Development and test problem assignments are frozen in the provenance manifest but have not yet been processed into example exports.

Motivation

Datasets built by asking a language model to invent a false or misleading user claim risk circularity: the claim may be labeled false without any concrete source showing why it is false. This dataset instead starts from narrow, inspectable literature evidence. A claim is admitted as misleading only when linked evidence directly contradicts it through a result, counterexample, assumption mismatch, impossibility statement, or explicit scope limitation. The fact that a research question remains open is not, by itself, treated as counterevidence.

The dataset also includes literature-supported, partially supported, speculative, and neutral examples. These controls are important because a system that simply rejects every user claim would perform poorly on the broader label set despite appearing robust on misleading examples alone.

Construction at a glance

The dataset follows an evidence-first construction order:

Open research problem
    ↓
Find relevant results in the literature
    ↓
Extract precise theorems, facts, assumptions, counterexamples, or limitations
    ↓
Verify the source, passage, locator, scope, and entailment
    ↓
Construct researcher priors using only the promoted evidence
    ↓
Validate that each prior has the correct relationship to its linked evidence

In other words, the pipeline does not begin by asking a language model to invent a misleading prior and then generate a justification for it. It first collects narrow, inspectable literature results related to the open problem and its unresolved target. Only after those results pass a separate verification stage are they used to construct priors.

For example, suppose a verified literature passage establishes:

A convergence theorem holds only under assumption A.

The same evidence can support several controlled prior types:

• —Misleading: “The method should converge without assumption A.”
• —Supported: “Under assumption A, this theorem gives us a useful convergence result.”
• —Partially supported: “The theorem gives convergence generally,” where the supported core is retained but the missing assumption must be restored.
• —Speculative: “Perhaps assumption A can be weakened; we should investigate whether the proof extends.”
• —Neutral: “What approaches might help analyze convergence for this problem?”

The extracted evidence is not assumed to solve the open problem. It may instead describe a known special case, a necessary assumption, a lower bound, an impossibility result, a counterexample, a limitation of an existing method, or established surrounding theory. The prior generator receives only promoted evidence cards, and a separate validation stage checks whether the assigned label—contradicted, supported, qualified, unsettled, or neutral—follows from that evidence.

Source material

The problem seeds originate from SolveAll, a catalog of open problems. Each canonical problem record retains its SolveAll URL, title, category assignments, problem statement, and a separately represented unresolved target.

The literature layer is assembled from source registries associated with each problem and, when needed, additional sources discovered from primary literature, official proceedings pages, publisher pages, or arXiv records. Each source record may contain authors, year, venue, DOI, arXiv identifier, URL, document version, and verification notes.

The repository does not assert ownership over SolveAll problem statements or quoted literature passages. Those materials remain subject to the terms and rights of their original sources.

Label ontology

The epistemic_label describes the relationship between the user's asserted claim and the verified evidence available for that problem. It is distinct from framing_strength, which records how strongly the user presents the claim.

Claim-bearing examples use weak or strong framing. Neutral examples use neutral framing and contain neither an asserted_claim nor an evidence link. The posture annotations indicate the evidence-consistent high-level response; they are not complete reference answers.

Dataset construction

1. Frozen problem splits

The 121 source problems are assigned once to category-aware train, development, and test splits using seed 20260721. The split manifest records the original source-file hash and the complete assignment list so that later pipeline runs cannot silently move problems across splits.

2. Canonical problem records and source registries

Each source problem is normalized into a schema containing a stable ID, slug, split, title, URL, categories, statement, unresolved target, and literature source registry. The unresolved target is represented separately to prevent a known surrounding result from being confused with a solution to the open problem itself.

3. Evidence discovery

Evidence discovery searches the inspectable literature for narrow results, counterexamples, impossibility statements, required assumptions, or explicit limitations relevant to each problem. Each proposed evidence card contains:

• —a precise reference_statement;
• —the conditions under which that statement holds;
• —at least one short passage from a source;
• —a page, section, theorem, proposition, equation, or paragraph locator; and
• —stable bibliographic identifiers when available.

Search-result snippets and claims inferred only from the absence of a result are not accepted as evidence passages. Evidence drafts remain pending until a separate verification stage passes them.

4. Evidence verification and repair

Evidence verification uses a separate fresh call and verifier prompt to check source identity, passage fidelity, locator accuracy, entailment, and scope. Passage text is normalized and bound to a SHA-256 hash. Evidence cards can be rejected, repaired from verifier feedback, and resubmitted, but repaired cards must pass verification again before promotion. In the current snapshot, this separation is procedural rather than model-level: both the research and verification roles used gpt-5.6-luna.

All 419 promoted evidence cards in the current completed problem records carry the status machine_verified. This status means that they passed the structured automated verification policy; it does not mean that every card has been reviewed by a human subject-matter expert.

5. Evidence-conditioned prior construction

Candidate user priors are constructed only from promoted evidence cards. Claim-bearing candidates must identify their exact evidence links, use the appropriate evidence relation, remain self-contained, distinguish established results from the unresolved target, and avoid invented quotations, citations, theorem names, or numerical details. If the evidence cannot support a requested label, the candidate is omitted rather than filled with a fabricated example.

Misleading candidates are intended to represent subtle and plausible research mistakes rather than obviously false statements. Supported candidates act as controls against indiscriminate disagreement. Partially supported candidates must identify the missing qualification, while speculative candidates must leave room for investigation and uncertainty.

6. Prior validation and repair

A separate validation pass checks each candidate for:

• —alignment between the claim and linked evidence;
• —correctness of the epistemic label and response posture;
• —self-containment;
• —naturalness as a researcher message;
• —absence of unsupported new factual details;
• —separation between established literature and the unresolved target; and
• —semantic duplication within the candidate pool.

Acceptance is derived from the individual validation criteria and duplicate status rather than trusting a free-form overall verdict. Failed or missing labels may be regenerated from validator feedback, but repaired candidates must receive a fresh validation call again.

7. Completeness and lineage gates

The canonical export includes only problems having at least one accepted example of all five epistemic labels. Every exported row must trace to the current evidence content, construction prompt version, validation prompt version, and validation input. The sibling manifest records hashes, required and observed prompt versions, completeness, and any lineage issues. The current export reports both mixture_complete: true and lineage_complete: true.

Automation disclosure

Evidence discovery, evidence repair, evidence verification, prior drafting, and prior validation use structured model-assisted workflows. The current provenance artifacts record gpt-5.6-luna for every one of those roles. Generation and verification use separate prompts and fresh calls, and a research call cannot promote its own pending evidence; however, the current release does not provide cross-model or human-independent verification. The dataset remains machine-generated and machine-verified and may contain errors. Users should inspect the cited primary sources for high-stakes or publication claims.

Snapshot statistics

Canonical label distribution

Framing distribution

Category coverage

Problems may have more than one category, so category counts exceed the number of examples.

Evidence links

The canonical export contains 1,236 evidence links across 893 claim-bearing rows. The relation counts are:

Some rows link more than one evidence card. The canonical export references 406 distinct problem-scoped evidence IDs; the completed problem records retain 419 promoted cards in total, including cards not selected by an accepted prior.

Repository contents

The recommended Hugging Face layout is:

.
├── README.md
├── data/
│   └── train.jsonl
├── source_seeds/
│   ├── problems.jsonl
│   ├── json/
│   └── md/
└── provenance/
    ├── manifests/
    ├── problems/
    ├── evidence_drafts/
    ├── evidence_verifications/
    ├── evidence_repaired_drafts/
    ├── evidence_repair_verifications/
    ├── verified_problems/
    ├── prior_drafts/
    ├── prior_validations/
    ├── prior_repairs/
    ├── validated_problems/
    └── exports/

data/train.jsonl is the canonical, lineage-complete export. source_seeds preserves the original 121-problem scrape as a combined JSONL file and per-problem JSON and Markdown files. The provenance/problems directory contains normalized records for all 121 frozen problems. Evidence, prior, validation, and completed-export artifacts are currently populated for the 85 train problems; 18 development and 18 test assignments remain frozen for a future release.

The provenance exports currently include:

The 800-row comparison mixture contains 320 misleading, 200 supported, 160 partially supported, 80 speculative, and 40 neutral examples. It is a derived view and should not be confused with the canonical 978-row export.

Canonical row schema

Each line in data/train.jsonl is a JSON object with the following fields:

Embedded evidence-card schema

Each embedded evidence card contains:

• —evidence_id: stable ID scoped to the problem;
• —reference_statement: the narrow fact licensed by the evidence;
• —scope_conditions: assumptions and limitations;
• —passages: source ID, locator, passage text, and normalized-text SHA-256;
• —verification_status; and
• —verification_notes.

Full bibliographic metadata for the passage source_id is available in the matching provenance/validated_problems/<seed_id>.json record.

Loading the dataset

Load the canonical public dataset with:

from datasets import load_dataset

dataset = load_dataset(
    "PranathReddy/solveall-literature-priors",
    data_files="data/train.jsonl",
    split="train",
)

print(dataset)
print(dataset.features)
print(dataset[0]["epistemic_label"])

To work directly with the nested evidence:

row = dataset[0]

for link in row["evidence_links"]:
    print(link["evidence_id"], link["relation"])

for card in row["evidence_cards"]:
    print(card["reference_statement"])
    for passage in card["passages"]:
        print(passage["source_id"], passage["locator"])

Quality controls

The current snapshot enforces the following structural guarantees:

• —every canonical problem has at least one accepted example of every label;
• —every claim-bearing prior has an asserted claim, reference position, neutral probe, and at least one evidence link;
• —every misleading prior has a contradiction link;
• —every supported prior has a support link;
• —every partially supported prior has a qualification link;
• —every speculative prior has a context or qualification link;
• —neutral priors have no asserted claim or evidence link;
• —embedded evidence passages have stable normalized-text hashes;
• —promoted evidence cards use verified sources;
• —internal source, evidence, and prior references resolve within each problem;
• —duplicate IDs are rejected within their problem scope;
• —canonical rows come only from five-label-complete problems; and
• —the export manifest reports complete, current lineage.

These are dataset-construction guarantees, not guarantees that every literature interpretation or annotation is correct.

Intended uses

Appropriate uses include:

• —studying evidence-sensitive responses to user framing;
• —evaluating agreement, disagreement, qualification, and uncertainty across controlled epistemic conditions;
• —analyzing how confidence of presentation interacts with evidence status;
• —developing claim-verification and citation-grounding methods;
• —auditing behavior on open-ended mathematical research discussions; and
• —studying failure modes in model-assisted scientific reasoning.

Uses requiring caution

This dataset should not be treated as:

• —an authoritative resolution of any SolveAll open problem;
• —a substitute for reading or citing the original literature;
• —a benchmark of formal proof correctness;
• —evidence that a claim is universally true or false outside the card's scope;
• —a source of legal, medical, financial, or other professional advice; or
• —a fully human-verified mathematical knowledge base.

Limitations and known risks

1. 1.Automated verification. Evidence and label checks are structured but machine-based. Subtle mathematical errors can survive automated review.
2. 2.Evidence coverage is selective. Cards capture narrow results relevant to constructing examples; they are not systematic literature reviews.
3. 3.Evidence-relative labels. Labels describe the relationship to the supplied verified evidence and its scope, not an unrestricted claim about all existing or future literature.
4. 4.Temporal staleness. Open problems and the literature can change after the recorded source version or access date.
5. 5.Domain imbalance. Mathematical statistics, learning theory, optimization, information theory, and probability dominate the snapshot.
6. 6.Single-language presentation. Prompts and annotations are in English.
7. 7.Synthetic researcher framing. User priors are constructed examples, even when they are designed to sound natural and are grounded in real literature.
8. 8.Current split coverage. Only the 85-problem train split has completed exported examples. Development and test assignments exist but are not part of this release.
9. 9.Shared model across roles. Research, repair, verification, generation, and validation used the same model family in the current snapshot. Separate prompts and calls reduce direct self-promotion but do not provide cross-model independence.
10. 10.Quoted source text. Short passages may be subject to source-specific copyright and usage terms.

Privacy and sensitive information

The dataset concerns public research problems and scholarly sources and is not designed to contain personal or confidential information. The artifact tree should nevertheless be scanned for credentials and unintended metadata before each upload. API credentials and local environment files are not dataset artifacts and must never be included.

Versioning and reproducibility

Dataset releases are identified by their Hugging Face repository revision and the schema_version field. The current schema version is 2.0.

The provenance manifest stores source-record hashes, construction metadata, validation lineage, and completeness checks. Internal pipeline identifiers are retained in the provenance artifacts for exact reproducibility but are not used as part of the public dataset name or release description. Consumers who need exact reproducibility should retain the manifest alongside the JSONL files.

Citation and attribution

No formal dataset citation has been assigned yet. When referencing an example, retain its seed_id, prior_id, SolveAll URL, evidence source metadata, and passage locator. Cite the original literature for substantive mathematical claims and acknowledge SolveAll as the source of the open-problem records.