Datasetpublic

spadeMIA/pmc_finetune_corpus_1024-2040_tokens

PMC 1024-2040 Biomedical Fine-Tuning Corpus Summary This is a cleaned biomedical long-text corpus for autoregressive language-model fine-tuning and held-out evaluation. split rows role train 10,000 fine-tuning test 1,000 held-out evaluation The public schema is text-only: text: string No PMCID, date, license, URL, or provenance fields are included in the public dataset files. Token Contract The corpus is built for… See the full description on the dataset page: https://huggingface.co/datasets/spadeMIA/pmc_finetune_corpus_1024-2040_tokens.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes49downloads

Dataset Card

PMC 1024-2040 Biomedical Fine-Tuning Corpus

Summary

This is a cleaned biomedical long-text corpus for autoregressive language-model fine-tuning and held-out evaluation.

split	rows	role
train	10,000	fine-tuning
test	1,000	held-out evaluation

The public schema is text-only:

text

text: string

No PMCID, date, license, URL, or provenance fields are included in the public dataset files.

Token Contract

The corpus is built for EleutherAI/pythia-2.8b.

Token counts were computed with:

python

tokenizer.encode(text, add_special_tokens=False)

requirement	value
stored text minimum	1024 tokens
stored text maximum	2039 tokens
EOS token budget	+1 token
maximum sequence budget	2040 tokens
EOS stored inside text	no

Final token-count statistics:

stat	train	test
min	1025	1048
mean	1953.9	1950.8
median	2002	2000
max	2039	2039
p1	1228.0	1185.8
p5	1640.0	1644.0
p25	1966.0	1960.0
p75	2021.0	2021.0
p95	2035.0	2034.0
p99	2039.0	2039.0

Source

Source dataset:

text

common-pile/pubmed_filtered

Rows were filtered to:

filter	requirement
publication date	publication_date >= 2020-09-01
language	English
token length	at least 1024 Pythia tokens after cleaning
license class	CC-BY, CC-BY-SA, or CC0

The Hugging Face metadata uses license: other because the corpus contains a mixture of CC-BY, CC-BY-SA, and CC0 rows.

Cleaning

Cleaning was deterministic and designed to remove PMC serialization artifacts without damaging biomedical prose, chemistry, units, or real math notation.

Removed or normalized artifact classes include:

artifact class	example
Pandoc anchors	`{#sec1}`
escaped citation groups	`\[[@bib1]\]`
numeric bracket citations	`\[1, 2\]`
bare citation markers	`[@CR40]`
empty citation parentheses	`()`, `(;)`, `(--)`
empty superscript citation shells	`^^`, `^,^`, `^--^`
cross-reference links	`[Figure 1](#fig1){ref-type="fig"}`
LaTeX document wrappers	`\documentclass`, `\begin{document}`
raw HTML/XML tags	`<p>`, `</italic>`
Word field-code remnants	`MERGEFORMAT`
confirmed escaped literal punctuation	`\*`, `\_`, `\#`
markdown image placeholders	`![](fx2.gif)`
generic markdown links	`[label](https://example.org)`
Pandoc smallcaps spans	`[TEXT]{.smallcaps}`
orphan bracket shells	`[[ word`, `[]`
dense markdown-table pipes	`	cell	cell	`

Ambiguous doubled backslashes and escaped parentheses are deliberately not auto-stripped because they can be legitimate LaTeX, math, or prose notation.

Truncation

Each selected row is a deterministic prefix of the cleaned parent document. The pipeline uses a three-tier boundary hierarchy:

boundary type	train	test
paragraph	8	1
sentence	9965	997
token-boundary cutoff	27	2

Processing Summary

stage	rows
rows scanned	27,000
eligible after filtering	24,846
unique eligible after deduplication	24,843
selected train rows	10,000
selected test rows	1,000
unused eligible rows	13,843

Date Distribution

publication year	train	test
2020	1142	109
2021	2307	234
2022	2538	248
2023	3165	323
2024	848	86

Minimum selected publication date:

text

2020-09-01

License Distribution

license class	train	test
CC-BY	9962	996
CC-BY-SA	6	2
CC0	32	2

Verification

The local review notebook re-scans every exported row. All strict artifact detectors pass with zero affected rows in both train and test:

strict artifact class	train	test
residual pandoc anchors	0	0
escaped-bracket citations	0	0
numeric bracket citations	0	0
dash-only citation remnants	0	0
empty citation parentheses	0	0
empty superscript citation shells	0	0
LaTeX document wrappers	0	0
raw HTML/XML tags	0	0
cross-reference links	0	0
bare citation markers	0	0
Word MERGEFORMAT artifacts	0	0
escaped literal punctuation	0	0
markdown image placeholders	0	0
generic markdown links	0	0
Pandoc smallcaps markup	0	0
orphan bracket shells	0	0
dense markdown-table pipes	0	0

Ordered sequence hashes over the public text column:

text

train_text_sequence_sha256 = 3f5767c782aca43da42cce8b456afb1e95e100b73c0abf4d4406e7d3b273338a
test_text_sequence_sha256  = e5207234ecbd3e80a41348a10da42a6c3ad36bba1e3c612e70f3dbbded77d37d

Reproducibility

field	value
preprocessing notebook	`pmc_1024_2040_preprocessing.ipynb`
review notebook	`pmc_1024_2040_dataset_review.ipynb`
artifact repair notebook	`pmc_1024_2040_artifact_repair.ipynb`
preprocessing version	`pmc_1024_2040`
seed	42
tokenizer	`EleutherAI/pythia-2.8b`
final audit timestamp	`2026-07-30T21:23:35.624609+00:00`

Use this corpus only under the token and cleaning contract documented above.