cogbuji/medqa_corpus_en

Dataset Card for MedQA English Textbooks

Table of Contents

• —Dataset Description
• —Dataset Summary
• —Languages
• —Dataset Structure
• —Data Instances
• —Dataset Creation
• —Curation Rationale
• —Source Data
• —Annotations
• —Personal and Sensitive Information
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Additional Information
• —Dataset Curators
• —Licensing Information
• —Citation Information

Dataset Description

Dataset Summary

MedQA includes

prepared text materials from a total of 18 English medical textbooks that have been widely used by medical students and USMLE takers" [Jin, Di, et al. 2020].

This dataset is derived from this medical textbook content (those in English), providing subsets that coincide with Medical subspecialties for use in pre-training medical LLMs with gold standard domain text.

Languages

English

Dataset Structure

Data Instances

Records have the following structure

{"text": "The manifestations of acute intestinal obstruction depend on the nature of the underlying [..]", 
 "source": "textbooks/en/InternalMed_Harrison.txt"}

Dataset Creation

Curation Rationale

The MedQA dataset includes raw text corpus that is excluded from most of its derivations and their dataset loading scripts . This raw text is valuable for pre-training of medical LLMS.

Source Data

Initial Data Collection and Normalization

Langchain's RecursiveCharacterTextSplitter is used for chunking and the most commonly-appearing non-ASCII characters are replaced with readable equivalents. Chunks comprising less than 90% ASCII characters were excluded. The textbooks were then broken into separate subsets, indicated below along with the textbook source(s) they comprise:

• —Core Clinical Medicine (*coreclinical*_)
• —AnatomyGray.txt (1,736 records), FirstAidStep1.txt (489 records), FirstAidStep2.txt (800 records), ImmunologyJaneway.txt (2,996 records), InternalMedHarrison.txt (20,583 records), NeurologyAdams.txt (7,732 records), ObstentricsWilliams.txt (5,392 records), PathomaHusain.txt (280 records), PediatricsNelson.txt (2,575 records), and SurgerySchwartz.txt (7,803 records)
• —Basic Biology (*basicbiology*_)
• —BiochemistryLippincott.txt (1,193 records), CellBiologyAlberts.txt (4,275 records), HistologyRoss.txt (2,685 records), PathologyRobbins.txt (3,156 records), and PhysiologyLevy.txt (2,627 records)
• —Pharmacology (*pharmacology*)
• —Pharmacology_Katzung.txt (4,505 records)
• —Psychiatry (*psychiatry*)
• —Psichiatry_DSM-5.txt (2,414 records)

So, you can load the basic biology subset of the corpus via:

In [1]: import datasets
In [2]: ds = datasets.load_dataset('cogbuji/medqa_corpus_en', 'basic_biology')
Generating train split: 50386 examples [00:00, 92862.56 examples/s]
In [3]: ds 
Out[3]: 
DatasetDict({
    train: Dataset({
        features: ['text', 'source'],
        num_rows: 50386
    })
})