Datasetpublic

domofon/britannica-pretrain

britannica-pretrain Private pretrain documents from Encyclopaedia Britannica scans (Internet Archive OCR *_djvu.txt), indexed via biglam/britannica-illustrated-pages manifest. One Arrow row is one article. No chunking. Trainer reads text. <encyclopedia> <meta source="britannica" edition="11th" year="1911" /> <title>Elasticity</title> <body> ... </body> </encyclopedia> <|endoftext|> <|endoftext|> is SmolLM2-1.7B eos. OCR is noisy. Duplicate library copies were collapsed by… See the full description on the dataset page: https://huggingface.co/datasets/domofon/britannica-pretrain.

sourceHugging Faceotherupdated 23d agoView on Hugging Face

0likes63downloads

Dataset Card

britannica-pretrain

Private pretrain documents from Encyclopaedia Britannica scans (Internet Archive OCR *_djvu.txt), indexed via biglam/britannica-illustrated-pages manifest. One Arrow row is one article. No chunking. Trainer reads text.

<encyclopedia>
<meta source="britannica" edition="11th" year="1911" />
<title>Elasticity</title>
<body>
...
</body>
</encyclopedia>
<|endoftext|>

<|endoftext|> is SmolLM2-1.7B eos. OCR is noisy. Duplicate library copies were collapsed by edition+volume number when the number parsed; some near-duplicate volumes remain. Images were not included.

Sidecar: id, source, title, edition, year, char_len, ia_id.