jiviteshjn/hi-proverbs-cpt

Hindi Proverbs — Idiom-Tagged Continued-Pretraining Corpus

A 338K-document Hindi corpus (~1.4B tokens) for continued pretraining on cultural knowledge in figurative language, plus a structured dataset of 16,617 Hindi proverbs (लोकोक्तियाँ) with meanings, recovered via OCR-repair from a classic proverb dictionary. Each corpus document is natural Hindi text containing at least one proverb (matched including common surface variants), with an appended knowledge block listing every matched proverb and its meaning.

Built 2026-07-17/18 as part of the CultureInFigurativeLanguage project (companion to the Chinese corpus built from chengyu + mC4 zh).

Key numbers

Contents

• —data/tagged_*.json.gz — the corpus. One JSON per line: {"text": "<document>\n\nलोकोक्तियों के अर्थ:\n<proverb> — <meaning>...", "url", "timestamp", "source": "mc4-hi"|"indiccorp-hi"|"fineweb2-hi", "shard", "line", "matched_idioms": [<canonical proverbs>], "original_text_chars"}
• —idioms/idioms_hi_llm_formatted.jsonl — the proverb dataset: {"idiom", "index", "source_index", "output": {"idiom", "entities", "literal_meanings" (empty for Hindi), "figurative_meanings"}}
• —idioms/idiom_variants_hi.jsonl — common surface variants used for matching: {"canonical", "index", "variants": [...]} (see below).
• —idioms/cross_language_parallels_hi.jsonl — the dictionary's तुलनीय blocks, structured: parallels of each proverb in Punjabi, Bhojpuri, Sanskrit, Rajasthani, Marathi, and ~40 other languages (all in Devanagari as printed).
• —idioms/cross_references_hi.jsonl — 2,019 दे० variant-form pointers.
• —idioms/entries_raw.jsonl, idioms/repair_decisions_hi.jsonl, idioms/variant_decisions_hi.jsonl — full provenance: rule-segmented OCR entries and every LLM repair/variant decision.
• —stats/ — extraction/selection/tagging summaries and per-proverb kept counts.

How it was built

1. 1.Proverb dataset from an OCR'd dictionary. Source: A Comprehensive Dictionary of Hindi Proverbs (बृहत् हिंदी लोकोक्ति कोश, ed. Bholanath Tiwari & Nur Nabi Abbasi), Digital Library of India scan (archive.org: in.ernet.dli.2015.464150), whose metadata states "dc.rights: In Public Domain". The Tesseract OCR text was segmented by rules into 15,200 entries, then repaired and structured by an LLM (gpt-5.4-mini) instructed to fix OCR errors conservatively, split glued entries, and DROP entries it could not confidently reconstruct (1,298 dropped). Every decision is included.
2. 2.Surface-variant generation for recall. Dictionary headwords use archaic morphology that modern text does not (होय vs होता है); an LLM generated commonly-occurring variants (gender/number agreement, verb modernization, postpositions, spelling, attested short forms; the dictionary's own दे० cross-reference forms were OCR-repaired in the same pass). Mechanical anti-invention validation (word-overlap and length guards, ambiguity dedup) kept 14,432 of 26,001 proposed variants. Variants match in the corpus but always report the canonical proverb.
3. 3.Corpus extraction. Two sources scanned with Aho-Corasick matching over Devanagari-normalized text (indic-nlp-library core: nukta removal, nasal-conjunct→anusvara; plus chandrabindu folding and punctuation/ whitespace collapsing): the mC4 hi train split (18.5M docs), HuggingFaceFW/fineweb-2 hin_Deva (22.1M docs, Common Crawl 2013-2024, ODC-BY; ~47% of its matched docs were near-duplicates of mC4 content and were removed by cross-source dedup, the remainder being largely 2021-2024 content mC4 predates) and AI4Bharat's IndicCorp v2 hi (6.1B tokens of verified news crawls, CC-0). Quality gates: 150–100,000 chars, ≤20 distinct proverbs per doc, minimum pattern length.
4. 4.Cross-source dedup + selection. Exact (blake2b) + MinHash-LSH near-dedup globally across both sources (~13K duplicates removed, including mC4↔IndicCorp overlap), then rare-proverb-first capped selection.
5. 5.Tagging. Each document gets a लोकोक्तियों के अर्थ: block listing its proverbs with up to 2 deduplicated meanings from the dictionary.

Usage

from datasets import load_dataset
ds = load_dataset("jiviteshjn/hi-proverbs-cpt", data_files="data/*.json.gz", split="train")

hf download jiviteshjn/hi-proverbs-cpt --repo-type dataset --local-dir hi_proverbs_cpt

Documents average ~6K tokens (mC4 pages are long); sequence packing recommended.

Provenance & licenses

• —Corpus text: `allenai/c4` multilingual hi (ODC-BY; inherits Common Crawl content caveats) and `ai4bharat/IndicCorpV2` hi (CC-0, human-verified news sources).
• —Proverb dictionary: Digital Library of India scan declared In Public Domain (see above). OCR errors were repaired conservatively; residual OCR noise may remain — the decisions files document every transformation.
• —Compilation released under ODC-BY (the most restrictive component). Intended for research on cultural alignment of language models.

Known limitation: even with variant matching, only 2,124 of the 16,590 proverbs occur in web text — the dictionary's archaic/rural tail is absent from modern corpora. The proverb dataset itself covers all 16,617 entries regardless.