jknafou/TransCorpus-bio

TransCorpus-bio

TransCorpus-bio is a large-scale, parallel biomedical corpus consisting of PubMed abstracts (title + abstract), translated with the TransCorpus Toolkit using NLLB-200. It is designed to enable high-quality multi-lingual biomedical language modeling and downstream NLP research.

This dataset was restructured from five separate single-language repositories into one dataset with a config (tab in the dataset viewer) per language, and with each row carrying its source PMID.

Languages

Every config has the same row count and the same schema: pmid (int64) and text (string, title + abstract on one line). The pmid column is the original PubMed ID of the source English abstract, is identical across every language for the same document, and is unique within each config, so joining any two configs on pmid reconstructs an aligned parallel pair. Row order and row count were verified to match exactly across all five languages and the source PMID list before publishing.

Usage

Load the default (English) configuration:

from datasets import load_dataset

dataset = load_dataset("jknafou/TransCorpus-bio", split="train")
print(dataset[0])
# {'pmid': 21, 'text': "[Biochemical studies on camomile components/III. ..."}

Load a specific language with name:

from datasets import load_dataset

dataset_fr = load_dataset("jknafou/TransCorpus-bio", name="fra_Latn", split="train")
print(dataset_fr[0])

Join two languages on pmid to get an aligned parallel pair:

from datasets import load_dataset

en = load_dataset("jknafou/TransCorpus-bio", name="eng_Latn", split="train")
es = load_dataset("jknafou/TransCorpus-bio", name="spa_Latn", split="train")
en_by_pmid = {row["pmid"]: row["text"] for row in en.select(range(1000))}
es_by_pmid = {row["pmid"]: row["text"] for row in es.select(range(1000))}
shared_pmid = next(iter(en_by_pmid.keys() & es_by_pmid.keys()))
print(en_by_pmid[shared_pmid])
print(es_by_pmid[shared_pmid])

Dataset Details

• —Source: PubMed abstracts (English)
• —Size: 21,567,136 abstracts per language, ~30GB of source text per language
• —Domain: Biomedical, clinical, life sciences
• —Identifiers: PubMed ID (pmid) per row, consistent across languages

Motivation

Non-English languages are low-resource languages for biomedical NLP, with limited availability of large, high-quality corpora. TransCorpus-bio bridges this gap by leveraging state-of-the-art neural machine translation to generate a massive, high-quality synthetic corpus, enabling robust pretraining and evaluation of biomedical language models in Spanish, French, German, and Hindi.

Benchmark Results in our French Experiment

TransBERT-bio-fr, pretrained on the French configuration of this corpus, achieves state-of-the-art results on the French biomedical benchmark DrBenchmark, outperforming both general-domain and previous domain-specific models on classification, NER, POS, and STS tasks. See TransBERT-bio-fr for details.

Why Synthetic Translation?

• —Scalable: Enables creation of large-scale corpora for any language with a strong MT system.
• —Effective: Supports state-of-the-art performance in downstream tasks.
• —Accessible: Makes domain-specific NLP feasible for any language.

Migration note

This dataset previously existed as five separate repositories (TransCorpus-bio, TransCorpus-bio-fr, TransCorpus-bio-es, TransCorpus-bio-de, TransCorpus-bio-hi), each a single unstructured text file with no PMID and no per-language tabs. Those repositories are kept in place (their file contents are unchanged, so existing citations and download links keep working) but are now marked as superseded in favour of this consolidated dataset.

Citation

If you use this corpus, please cite:

  @inproceedings{knafou-etal-2025-transbert,
      title = "{T}rans{BERT}: A Framework for Synthetic Translation in Domain-Specific Language Modeling",
      author = {Knafou, Julien  and
        Mottin, Luc  and
        Mottaz, Ana{\"i}s  and
        Flament, Alexandre  and
        Ruch, Patrick},
      editor = "Christodoulopoulos, Christos  and
        Chakraborty, Tanmoy  and
        Rose, Carolyn  and
        Peng, Violet",
      booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
      month = nov,
      year = "2025",
      address = "Suzhou, China",
      publisher = "Association for Computational Linguistics",
      url = "https://aclanthology.org/2025.findings-emnlp.1053/",
      doi = "10.18653/v1/2025.findings-emnlp.1053",
      pages = "19338--19354",
      ISBN = "979-8-89176-335-7",
      abstract = "The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs."
  }