jiviteshjn/hi-proverbs-cpt
Hindi Proverbs — Idiom-Tagged Continued-Pretraining Corpus A 338K-document Hindi corpus (~1.4B tokens) for continued pretraining on cultural knowledge in figurative language, plus a structured dataset of 16,617 Hindi proverbs (लोकोक्तियाँ) with meanings, recovered via OCR-repair from a classic proverb dictionary. Each corpus document is natural Hindi text containing at least one proverb (matched including common surface variants), with an appended knowledge block listing every… See the full description on the dataset page: https://huggingface.co/datasets/jiviteshjn/hi-proverbs-cpt.
Expand with fineweb-2 hin_Deva: 338K docs / ~1.37B tokens, coverage 1937->2124, cross-source deduped (part 5)
Expand with fineweb-2 hin_Deva: 338K docs / ~1.37B tokens, coverage 1937->2124, cross-source deduped (part 4)
Expand with fineweb-2 hin_Deva: 338K docs / ~1.37B tokens, coverage 1937->2124, cross-source deduped (part 3)
Expand with fineweb-2 hin_Deva: 338K docs / ~1.37B tokens, coverage 1937->2124, cross-source deduped (part 2)
Expand with fineweb-2 hin_Deva: 338K docs / ~1.37B tokens, coverage 1937->2124, cross-source deduped
Correct token estimate: ~1.1B (per-source measurement)
Refresh: variant-augmented matching (+45% corpus, coverage 1488->1937); add idiom_variants + variant decisions (part 5)
Refresh: variant-augmented matching (+45% corpus, coverage 1488->1937); add idiom_variants + variant decisions (part 4)
Refresh: variant-augmented matching (+45% corpus, coverage 1488->1937); add idiom_variants + variant decisions (part 3)
Refresh: variant-augmented matching (+45% corpus, coverage 1488->1937); add idiom_variants + variant decisions (part 2)
Refresh: variant-augmented matching (+45% corpus, coverage 1488->1937); add idiom_variants + variant decisions
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
initial commit