CoolFace
Datasetpublic

anuj1541/marathi-english-cidco-long-docs

Marathi-English CIDCO Long Documents & Paragraphs Dataset Parallel Marathi-to-English translation dataset extracted from CIDCO (City and Industrial Development Corporation of Maharashtra) government documents, resolutions, tenders, and town planning records. This version is specifically filtered to retain only long, fluent sentences and multi-sentence paragraphs, removing single-word table cells, short headers, numbers, and form labels. Dataset Splits Split… See the full description on the dataset page: https://huggingface.co/datasets/anuj1541/marathi-english-cidco-long-docs.

sourceHugging Faceotherupdated 16d agoView on Hugging Face
0likes76downloads
Dataset Card

Marathi-English CIDCO Long Documents & Paragraphs Dataset

Parallel Marathi-to-English translation dataset extracted from CIDCO (City and Industrial Development Corporation of Maharashtra) government documents, resolutions, tenders, and town planning records.

This version is specifically filtered to retain only long, fluent sentences and multi-sentence paragraphs, removing single-word table cells, short headers, numbers, and form labels.

Dataset Splits

SplitRowsMarathi Mean WordsEnglish Mean WordsDescription
train56,352~38 words~41 wordsFiltered training split
validation1,617~38 words~41 wordsValidation split for model checkpointing
test788~38 words~41 wordsHeld-out test split for MT evaluation

Filtering Criteria

  • —Minimum Marathi Words: $\ge 20$
  • —Minimum English Words: $\ge 20$
  • —Minimum Marathi Characters: $\ge 100$
  • —Minimum English Characters: $\ge 100$
  • —Length Ratio: $0.4 \le ext{words(MR)} / ext{words(EN)} \le 2.5$

Usage

python
from datasets import load_dataset

dataset = load_dataset("anuj1541/marathi-english-cidco-long-docs")
print(dataset)