jhdlee/wiki-fact

Wiki Fact

The all_articles configuration of jhdlee/wiki-fact contains 20,049 complete Wikipedia-derived articles retained by the frozen v4 mechanical screen: 14,514 in cohort A and 5,535 in cohort B, across 19 topics. It is a reusable source pool for research on learning factual information from text. Future selections can be released as additional configurations, leaving all_articles membership fixed.

The train split is a storage convention for this unsplit pool. It does not assign an evaluation protocol. Cohorts are recorded in the cohort column:

Both cohorts use the same August 2025 published text snapshot, from HuggingFaceFW/finewiki, configuration en, split train, revision 8bd13e72e6a002407649b3e898535f42ceb1aeb9. Cohort assignment follows the classified central event, first release or first opening date, rather than the page revision timestamp. Older background and later updates remain in the articles, so the cohorts do not guarantee separation of facts across time.

text is the exact authenticated CPT string used by the screening run. The earlier CPT formatting step replaced FineWiki's generated title heading with the stripped title and two newline characters, and removed only outer whitespace around the complete body. Publication does not repeat that formatting step, normalize Unicode, repair text, remove the known audit flags, or truncate articles.

Articles contain 512–3,584 full-CPT tokens, inclusive, under Qwen/Qwen2.5-0.5B-Instruct at revision 7ae557604adf67be50417f59c2c2f167def9a775, with no added special tokens and no truncation. Existing authenticated measurements total 30,121,172 tokens; mean length is approximately 1,502 tokens. These are tokenizer-specific full-document counts, not narrative-only counts.

Selection and inspection

Discovery first used the short-document length filter and target-year mentions in title/text/supplied content metadata, followed by low-reasoning classification and xhigh refinement. This prioritization does not establish exhaustive coverage of every eligible event in FineWiki. The other_event_or_release category was removed before this working pool. The input working pool contained 21,640 articles assigned to A/B by the historical xhigh_model_gold source-classification process. The frozen short-clean-only-screen-v4.0 policy retained 20,049 and excluded 1,591, with complete processing coverage and zero processing errors. Its policy hash is 4028a6c36be386661ce792e6cb144782c3fd1d7178e570215e173ab31ea3b445.

V4 screens for corrupted characters, unmistakable Wikimedia/rendering/layout residue, wholly empty structures and repeated artifact-only lines. It separately excludes non-Latin letters anywhere in titles or complete CPT text, including code and mathematics, and the business_events, public_health_events and scientific_and_astronomical_events topics. Accented Latin letters, combining marks, symbols and ordinary mathematical notation are permitted. Valid placeholders, populated sparse tables, substantive header-only tables and ordinary Markdown separators are retained.

A subsequent direct inspection read the complete titles and CPT texts of 200 fresh survivors selected using seed 42 with cohort/topic/length stratification. Reviewers found 195 without a clear mechanical defect and five with residual defects. All five remain in this release as requested, with their audit metadata. They comprise three escaped complete tables embedded in cells, one leaked tooltip label and one article with portrait-image filenames in table cells. Uninspected rows must not be interpreted as having passed direct inspection.

The 5/200 fraction describes this stratified sample; it is not a corpus-wide defect-rate estimate. The screen and inspection do not certify factual correctness, complete extraction, English language, deduplication of facts, independence between articles, or previous model ignorance. Most sampled articles received one direct reader, with adjudication of flagged evidence. Historical model classifications and Codex inspections are not human review approvals.

Data and provenance

Each article includes its stable article_id, title, complete text, cohort, topic, token count, exact text hash, original page/revision metadata and source locators. Original published-text and record hashes remain separate from the CPT text hash. Qualifying dates retain the recorded text and date basis without inventing exact interval endpoints. Mechanical screening status and subsequent direct-inspection status are separate fields.

The main columns are:

direct_audit_status is no_clear_defect for 195 articles, clear_mechanical_defect for five, and not_in_fresh200_audit for 19,849. The last status does not erase any historical inspections. Non-string infobox metadata is JSON encoded; strings are preserved. Metadata, including infoboxes, is not inserted into text, and the Latin-letter screen was applied to the title and CPT text rather than every metadata field.

The release manifest binds exact retained membership, source archive, pinned FineWiki snapshot, length-tokenizer identity, screening policy, direct-inspection metadata and Parquet checksums. The publication preserves all 20,049 source IDs and CPT hashes, including the five flagged articles. Resolved source-classification evidence is in metadata/classification_evidence.jsonl.gz; raw generation reasoning and execution logs are excluded.

from datasets import load_dataset

pool = load_dataset("jhdlee/wiki-fact", "all_articles", split="train")
cohort_a = pool.filter(lambda row: row["cohort"] == "A")

Pin a dataset commit through revision= when reproducing an experiment.

Adding future subsets

Keep all_articles as the immutable seed configuration. Add a new named configuration and its own data files for each future selection. Record the parent repository commit and pool_release_manifest.json hash, the parent configuration (all_articles), selected article IDs and text hashes, selection rules, and any random seed in that subset's manifest. Reuse the existing IDs and unchanged text; do not overwrite or resample the base configuration. No subsets or QAs are included in this initial release.

License and attribution

The pinned FineWiki documentation states that its processed release is an adaptation of Wikipedia text licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). This release follows that license. Credit Wikipedia contributors and Guilherme Penedo / FineWiki (2025), retain source attribution and license notices, and indicate the selection, metadata additions and documented CPT serialization when redistributing this adaptation.

Per-row source and revision URLs identify the original articles. Contributor histories can be accessed through https://en.wikipedia.org/w/index.php?curid=PAGE_ID&action=history. The upstream notice also records the underlying Wikipedia text's GFDL availability; this does not assert a separate GFDL license for FineWiki's processed adaptation. See the Wikipedia dump licensing notice.

@dataset{penedo2025finewiki,
  author = {Guilherme Penedo},
  title = {FineWiki},
  year = {2025},
  publisher = {Hugging Face Datasets},
  url = {https://huggingface.co/datasets/HuggingFaceFW/finewiki}
}

Filtered articles derivative

The filtered_articles configuration is a section-filtered derivative of all_articles at 944ad96042848830b1efb15d272bd1a3fdb70d05. The original configuration and its files are unchanged; the preceding source-pool description applies to all_articles.

All 20,049 original articles were processed; 10,489 were excluded because fewer than 512 full-CPT tokens remained, including empty results. Counts use Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775 without special tokens or truncation.

The frozen mechanical policy removes a section's immediate body if it contains a table recognized by the pinned Markdown/FineWiki parser, removes empty sections, and removes an exact listed set of reference/navigation headings and descendants. Code examples are protected. Necessary ancestor headings survive with retained children. Short sections are preserved; no minimum section length, mandatory lead, new extraction-defect checks, or historical-audit gates are applied. Complete details and hashes are in metadata/filtered_articles/policy.json and the release manifest. The minimum article length is inclusive: 512 tokens passes.

text is formed exclusively by concatenating retained original Unicode spans; text_sha256 and cpt_tokens describe that derivative. original_text_sha256, original_cpt_tokens, parent_dataset_revision, parent_dataset_config, section_filter_policy, section_filter_policy_sha256, section_filter_min_cpt_tokens, removed_section_count, retained_spans_json, and text_modified record the transformation. All other original fields are preserved. Inherited source/classification/infobox/audit metadata describe the original full article and may reference removed passages. They were not rechecked against the shortened text. retained_spans_json uses end-exclusive Unicode character offsets in the original CPT text.

This is deterministic formatting and length filtering, not factual verification or a guarantee of meaningful prose, coherent cross-section references, or detection of every malformed table. Existing source-screening limitations remain. Both cohorts share the published August 2025 text snapshot. The single train split is storage only and does not establish a training/evaluation partition.

Section deletion is an additional adaptation, licensed under CC BY-SA 4.0. Credit Wikipedia contributors and Guilherme Penedo / FineWiki (2025), preserve the per-row source/revision attribution and license notices, and indicate the changes. The parent license file remains unchanged; its statement about unchanged text describes the original all_articles release, not this documented derivative.

from datasets import load_dataset
filtered = load_dataset("jhdlee/wiki-fact", "filtered_articles", split="train")

Pin the new release commit for reproducibility. No QAs or training outcomes are introduced by this configuration.