CoolFace
Datasetpublic

DimitarV/bulgarian-medical-cpt-100m

Bulgarian text for MOSS continued pretraining Exactly 100 million training tokens: 10M medical and 90M general Bulgarian. An additional 100,000 tokens are provided for validation (50k per source). No audio, instruction-response pairs, or generated answers. No model training has been performed as part of this dataset build. Medical data Exactly 10,000,000 training tokens and 50,000 additional validation tokens, including one <|im_end|> EOS per record. Extracted… See the full description on the dataset page: https://huggingface.co/datasets/DimitarV/bulgarian-medical-cpt-100m.

sourceHugging Faceotherupdated 16d agoView on Hugging Face
0likes88downloads
Dataset Card

Bulgarian text for MOSS continued pretraining

Exactly 100 million training tokens: 10M medical and 90M general Bulgarian. An additional 100,000 tokens are provided for validation (50k per source). No audio, instruction-response pairs, or generated answers. No model training has been performed as part of this dataset build.

Medical data

Exactly 10,000,000 training tokens and 50,000 additional validation tokens, including one <|im_end|> EOS per record. Extracted Bulgarian EMA product information: SmPC sections and patient leaflets. See the subset summaries for training record and section counts. Validation contains the same 48 medical records as the 10M pilot. Products are assigned to validation by a deterministic hash before sampling; no product appears in both splits. Closely related medicines may share language across products: this is a development split, not an independent medical benchmark.

EMA is the source of the medical material. Each row preserves the product URL, product identifier, section and section type. EMA's legal notice allows reuse with attribution, subject to third-party exceptions. This dataset does not relicense the source material or claim it is public domain. No affiliation with or endorsement by EMA is implied. The PDFs were previously downloaded locally; their extraction may retain layout artifacts. Not clinician-reviewed.

Tokenization and training use

Tokenizer: OpenMOSS-Team/MOSS-Transcribe-Diarize, revision 704aa4a9c304e8520be88901e0d1960158ef5b15. EOS ID: 151645. input_ids are the authoritative tokenized sequences; num_tokens includes EOS. text is their decoded content without the appended EOS. Records contain at most 2,048 tokens; long sections/documents are split at token boundaries without splitting UTF-8 characters. The last record may be shortened to meet the budget. These are independent causal-LM sequences; when batching, mask padding loss and use an attention mask. Do not append another EOS to input_ids. Retokenizing or changing the tokenizer will change the published counts. No invented speaker labels, timestamps, or audio placeholders are included.

Preparation and limitations

NFC normalization, soft-hyphen/control-character removal, whitespace cleanup, Cyrillic-majority and minimum-length filtering, exact substantial-paragraph deduplication, deterministic sampling. Source text is not LLM-rewritten. Deduplication includes normalized exact matches and candidate-based near-duplicate document detection: bottom four 5-word-shingle hash keys identify candidates; exact shingle Jaccard >=0.90 rejects them. Candidate retrieval can miss duplicates; this is not exhaustive pairwise deduplication. Some repeated regulatory language and tables remain. MOSS's original pretraining data may already contain these public sources; no novelty claim is made.

This is a CPT experiment corpus, not a clinical knowledge benchmark or a set of validated clinical recommendations. Audio ASR/diarization evaluation remains necessary to establish whether text CPT helps MOSS.

General data

Source: HuggingFaceFW/fineweb-2, Bulgarian bul_Cyrl, revision af9c13333eb981300149d5ca60a8e9d659b276b9. Train and validation are sampled from separate upstream train and test shards. URLs and shard provenance are retained. Dataset license: ODC-By-1.0; original website rights are not replaced by that collection license. See https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 . This is filtered web prose, not medically verified content. Selection excludes obvious email contacts, several classified/directory domains, short or highly fragmented pages, excessive uppercase and repeated lines. It is a deterministic bounded sample across six corpus-spread Bulgarian training shards, each contributing 15M tokens; sampling.json records the source shards. It is not a representative random sample of the entire web corpus. No comprehensive PII removal is claimed.

Loading

python
from datasets import load_dataset
corpus = load_dataset("DimitarV/bulgarian-medical-cpt-100m", "combined")
medical = load_dataset("DimitarV/bulgarian-medical-cpt-100m", "medical")
general = load_dataset("DimitarV/bulgarian-medical-cpt-100m", "general")

Shuffle the combined training split before training; files are separated by source for reproducible mixing and ablations. The 90:10 ratio is by supervised tokens including EOS, not by record count. Use ordinary causal next-token loss, without a chat template. Decoder-only training integration is separate work; the stock MOSS audio-only collator cannot ingest these records unchanged.

validation_report.json records token totals, decoding consistency, disjoint source documents/products, exact text overlap checks and local file hashes. Preparation scripts are included in preparation/. Near-duplicate and pretraining-contamination limitations remain as described above.

Relationship to the 10M pilot

This dataset overlaps the 10M training corpus and is not an independent benchmark. It copies the same 50k medical and 50k general validation tokens. All medical products assigned to the pilot holdout remain excluded from training, and general training uses only upstream training shards. Validation text seeds the deduplication index. Validation URLs and exact validation records are excluded. This is continued-pretraining data only; no model has been trained on it here.