doctolib-lab/finemed-fr

FineMed-fr

<center> <img src="assets/logo_finemed.png" width="100%" alt="FineMed-fr: A large-scale French medical corpus annotated along multiple quality axes"> </center>

<p align="center"> <a href="https://huggingface.co/blog/bofenghuang/doctobert-fr-release">🤗 Blog</a> | <a href="https://arxiv.org/abs/2606.22079">📄 Paper</a> | <a href="https://github.com/doctolib-lab/doctobert">💻 Code</a> | <a href="https://huggingface.co/collections/doctolib-lab/finemed-fr">🌐 FineMed</a> | <a href="https://huggingface.co/collections/doctolib-lab/doctobert-fr">🩺 DoctoBERT</a> </p>

📚 Introduction

FineMed-fr is a large, openly available corpus of French medical text for language-model pretraining: 21.1M documents and 19.2B words of real-world medical writing, annotated along several quality axes.

The corpus is drawn from three heterogeneous open-web sources (FineWeb-2, FinePDFs, and FineWiki), which together provide the scale, source diversity, and stylistic range that curated medical corpora often lack. We keep only the French medical content, then label every surviving document along three axes:

• —Subdomain: which of 15 medical subdomains the document belongs to, separating biomedical and clinical writing (e.g. scientific papers, clinical guidelines) from consumer-facing material (e.g. wellness blogs, commercial health pages).
• —Educational quality: how instructive the document is for medical education, scored 0–5 on an additive rubric adapted from FineWeb-Edu.
• —Medical-term density: the richness of medical terminology, measured as the fraction of characters that fall inside extracted medical-term spans.

We release the corpus unfiltered so you can set your own thresholds on the annotation columns to fit your task.

🆕 What's New

• —v1.0 (2026-06): first release.

🚀 How to Use

from datasets import load_dataset

ds = load_dataset("doctolib-lab/finemed-fr", split="train")  # fineweb-2 (default)
ds = load_dataset("doctolib-lab/finemed-fr", "finepdfs", split="train")
ds = load_dataset("doctolib-lab/finemed-fr", "finewiki", split="train")

Because the corpus is released unfiltered, downstream filtering is left to the user. For example, to retain only high-quality, term-dense documents:

filtered = ds.filter(
    lambda x: x["edu_quality_normalized_score"] >= 4 and x["medical_entity_density"] >= 0.10,
    num_proc=8,
)

🔧 Curation Pipeline

The three source corpora have already undergone standard LLM-pretraining curation upstream (language ID, heuristic quality filtering, deduplication), which we inherit as a quality baseline. Beyond this baseline, we apply two further steps:

1. 1.Medical prefiltering. Medical content constitutes only a small fraction of each source and is further diluted by commercial pages. We run a multilingual domain classifier (a DeBERTa-v3 covering 26 domains) over the first 512 tokens of each document and retain only those whose top-1 predicted label is Health, reducing each source to 5.3% of FineWeb-2, 7.7% of FinePDFs, and 1.5% of FineWiki (by document).
2. 2.Multi-axis annotation. Every retained document is then labeled by three dedicated lightweight annotators, each fine-tuned via two-stage knowledge distillation from LLM teachers (a smaller teacher providing high-volume supervision, followed by a larger teacher providing high-quality supervision):
3. 3.A subdomain classifier takes the document text and URL as input and predicts one of 15 medical subdomains;
4. 4.An educational-quality scorer takes the document text and regresses a 0–5 educational-quality score;
5. 5.A medical-entity extractor identifies medical-term spans, whose character coverage defines the medical-term density.

Distilling each annotator from its LLM teachers, rather than applying an LLM directly across the full corpus, reduces annotation cost by roughly an order of magnitude.

Subdomain. health_domain_classification_best_class is one of these 15 values:

Medical-term classes. medical_entities groups the extracted terms under these 8 keys (taxonomy adapted from UMLS):

Educational quality. edu_quality_normalized_score runs from 0 (not useful) to 5 (excellent) for medical education; edu_quality_score is the raw value before rounding. The exact rubric used to prompt the LLM annotators is in `edu_quality_annotation_prompt.txt`.

📊 Dataset Statistics

Each source is provided as a separate config. Per-source statistics:

<!-- Average annotation values per source:

Annotation values vary substantially across subdomains. Distribution of educational-quality scores across the 15 subdomains:

[image]

Distribution of medical-term density across the same subdomains:

[image]

Per-source versions of both plots are available in assets/.

<!-- Domain composition of each source (% of documents; Health is the retained fraction):

📋 Data Fields

All configs share these columns:

Source-specific provenance columns:

• —fineweb-2: dump, date, file_path, language, language_score, language_script, minhash_cluster_size, top_langs
• —finepdfs: dump, date, file_path, offset, token_count, language, page_average_lid, page_average_lid_score, full_doc_lid, full_doc_lid_score, per_page_languages, is_truncated, extractor, page_ends
• —finewiki: wikiname, page_id, title, date_modified, in_language, wikidata_id, bytes_html, wikitext, version, infoboxes, has_math

Example record. A full fineweb-2 row (provenance columns differ for the other configs):

{
  "text": "Attention L'actualité thérapeutique sur le VIH évolue rapidement ... Pneumopathie bactérienne chez les patients infectés par le VIH ...",
  "id": "<urn:uuid:2cc73ad5-d0ae-483c-8b59-78147734bcb8>",
  "dump": "CC-MAIN-2014-10",
  "url": "http://www.actions-traitements.org/spip.php?article1961",
  "date": "2014-03-10T06:59:32Z",
  "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2014-10/segments/.../CC-MAIN-...-00091-....warc.gz",
  "language": "fra",
  "language_score": 0.9974,
  "language_script": "Latn",
  "minhash_cluster_size": 7,
  "top_langs": "{}",
  "domain_classification_scores": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  "domain_classification_best_class": "Health",
  "domain_classification_best_score": 1.0,
  "num_words": 600,
  "health_domain_classification_scores": [0.0092, 0.0349, 0.832, 0.0, 0.0267, 0.0, 0.0001, 0.0001, 0.0001, 0.0, 0.0001, 0.0, 0.0043, 0.0933, 0.0],
  "health_domain_classification_best_class": "Clinical guidelines & pathways",
  "health_domain_classification_best_score": 0.832,
  "edu_quality_score": 4.75,
  "edu_quality_normalized_score": 5,
  "medical_entities": {
    "disease": ["pneumonie", "méningites", "fièvre", "sida", "..."],
    "drug": ["traitement antirétroviral"],
    "body_part": [], "medical_procedure": [], "molecular_marker": [],
    "clinical_device": [], "vital_function": [],
    "living_beings": ["Streptococcus pneumoniae", "Klebsiella pneumoniae", "VIH", "..."]
  },
  "medical_entity_density": 0.242
}

(a real FineWeb-2 row; `text`, `file_path`, and entity lists trimmed for display)

<!--

📖 Citation

@misc{doctobert2026,
  title         = {Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining},
  author        = {Huang, Bofeng and Sun, Jacques and Bouchacourt, Diane and Barascud, Nicolas and Fogel, Fajwel},
  year          = {2026},
  eprint        = {2606.XXXXX},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

-->

⚖️ Licensing

FineMed-fr inherits the licenses of its source datasets:

• —fineweb-2 and finepdfs: ODC-BY 1.0 (as in the upstream FineWeb releases)
• —finewiki: CC BY-SA 4.0 (derived from Wikipedia)

⚠️ Considerations

FineMed-fr consists of public text from the web, PDFs, and Wikipedia, restricted to medical content. As real-world web data, it may contain personal information, and medical pages may reference protected health information. All such content was already publicly accessible, and we did not remove or mask it. The corpus has not been clinically validated and does not constitute medical advice. Users handling personal or health data should perform de-identification before use.

🏛️ Acknowledgments

This work was granted access to the HPC resources of IDRIS (Jean Zay) under the allocations 2025-AD011016291 and 2026-A0200617487 made by GENCI.