joecwales/whiteglove-medical-medlineplus-2025

WhiteGlove Medical Knowledge Corpus

MedlinePlus 2025 — Spectral Curation Pipeline

Pipeline: WhiteGlove Spectral Curation | Domain: Medical | License: Public Domain (US Government)

Dataset Summary

A clean, deduplicated, semantically chunked medical knowledge corpus derived from the NIH MedlinePlus January 2025 ZIM archive. Produced by the WhiteGlove Spectral Curation Pipeline — an air-gapped, attribution-clean dataset factory built on SimHash-128 deduplication and semantic chunking.

Pipeline Architecture

Kiwix MedlinePlus .zim (1.8GB)
    │
    ▼ zim_extractor.py
Raw HTML shards → staging/ (12,297 pages)
    │
    ▼ rechunk_medical.py
HTML strip + boilerplate removal
500-word semantic chunks → shattered/ (33,685 shards)
    │
    ▼ export_training_set.py
SimHash-64 deduplication (Hamming ratio < 0.12)
Quality scoring (unique word ratio, punctuation density, noise patterns)
Train/validation split (95/5)
    │
    ▼
HF-compatible JSONL (train.jsonl, validation.jsonl)
+ manifest.json (full provenance)

Key properties:

• —Zero cloud dependency — entire pipeline runs air-gapped on consumer hardware
• —SimHash-128 deduplication — catches near-duplicates that exact-match dedup misses
• —Semantic chunking — 500-word windows preserve topical coherence vs arbitrary byte splits
• —Full provenance — every record traces back to source page, chunk index, and ZIM entry
• —Attribution-clean — MedlinePlus is public domain (US Government, NIH/NLM)

Temporal Dimension Note

The WhiteGlove pipeline supports time as an embedding dimension for dynamic corpora (e.g., versioned codebases, game state sequences, evolving documentation). Static corpora like MedlinePlus use a single timestamp. For time-aware use cases (behavioral game code, event logs, versioned APIs), the pipeline emits chunk_index + source_shard_id as positional anchors that downstream embedders can use to reconstruct temporal order within a document.

Record Schema

{
  "text": "...",
  "instruction": "Provide accurate medical information about: {title}",
  "output": "...",
  "id": "med_chunk_000042",
  "source": "MedlinePlus",
  "source_tag": "MedlinePlus-2025-01",
  "domain": "medical",
  "license": "public-domain",
  "title": "A1C: MedlinePlus",
  "path": "medlineplus.gov/lab-tests/a1c-test/",
  "source_shard_id": "med_000003",
  "chunk_index": 0,
  "total_chunks": 2,
  "word_count": 487,
  "quality_score": 0.6821,
  "simhash": 12389471923847,
  "schema_version": "whiteglove-v1"
}

Fields:

• —text — primary training signal (plain text, HTML stripped)
• —instruction / output — SFT format for supervised fine-tuning
• —quality_score — 0.0–1.0 heuristic (unique word ratio, punctuation density, noise penalty)
• —simhash — 64-bit SimHash signature for downstream dedup verification
• —source_shard_id — links back to original ZIM entry for provenance

Usage

from datasets import load_dataset

ds = load_dataset("joecwales/whiteglove-medical-medlineplus-2025")
print(ds["train"][0]["text"])

# Filter by quality
high_quality = ds["train"].filter(lambda x: x["quality_score"] > 0.65)

Fine-tuning (SFT format)

# The instruction/output fields are pre-formatted for SFT
# Compatible with TRL SFTTrainer, LLaMA Factory, Axolotl
ds = load_dataset("joecwales/whiteglove-medical-medlineplus-2025")

Curation Pipeline — Reproduce It

The full pipeline is open. To build your own domain pack from any Kiwix ZIM:

1. 1.Download any .zim from kiwix.org
2. 2.Run zim_extractor.py → raw HTML shards
3. 3.Run rechunk_medical.py (adaptable to any domain) → clean chunks
4. 4.Run export_training_set.py → deduplicated JSONL

Custom domains supported: Legal codes, Python documentation, Wikipedia subsets, government archives, technical manuals — anything available as a Kiwix ZIM.

WhiteGlove Pipeline

This dataset was produced by the WhiteGlove Spectral Curation Pipeline — a sovereign, air-gapped dataset factory. Core properties:

• —SimHash-128 deduplication (O(k) lookup, no vector database required)
• —Semantic landmark indexing for Faith-Less retrieval (238ms over 33k shards, offline)
• —Time-aware chunking for dynamic corpora (game code, versioned docs, event sequences)
• —Sector-agnostic: swap the ZIM, same pipeline, new domain

Embed once. Query forever. No cloud. No fine-tuning required.

Citation

@dataset{whiteglove_medical_2025,
  title     = {WhiteGlove Medical Knowledge Corpus — MedlinePlus 2025},
  author    = {Wales, Joe},
  year      = {2025},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/joecwales/whiteglove-medical-medlineplus-2025},
  note      = {Produced by the WhiteGlove Spectral Curation Pipeline.
               Source: NIH MedlinePlus (public domain).}
}

Source Attribution

MedlinePlus is produced by the National Library of Medicine (NLM), National Institutes of Health (NIH), US Department of Health and Human Services. As a work of the US Federal Government, it is in the public domain.

• —Source: medlineplus.gov
• —Archive: Kiwix medlineplus.gov_en_all_2025-01.zim