Datasetpublic

CodeIsAbstract/sanskrit-sandhi-samas-v3

Sanskrit Sandhi + Samas Boundary Dataset — V3 The canonical merged training dataset for the sanskrit-sandhi-boundary model: sentence-level external sandhi (corpus) + grammar-generated samas (compounds, all 7 types) in one file. Composition Source Rows Description sandhi_corpus 741,803 running-text word boundaries (external sandhi) from sanskrit-sandhi-boundaries-v2 samas 258,408 grammar-generated compounds (7 types, laukik + alaukik vigraha) from… See the full description on the dataset page: https://huggingface.co/datasets/CodeIsAbstract/sanskrit-sandhi-samas-v3.

sourceHugging Facemitupdated 28d agoView on Hugging Face

0likes40downloads

Dataset Card

Sanskrit Sandhi + Samas Boundary Dataset — V3

The canonical merged training dataset for the sanskrit-sandhi-boundary model: sentence-level external sandhi (corpus) + grammar-generated samas (compounds, all 7 types) in one file.

Composition

Source	Rows	Description
sandhi_corpus	741,803	running-text word boundaries (external sandhi) from `sanskrit-sandhi-boundaries-v2`
samas	258,408	grammar-generated compounds (7 types, laukik + alaukik vigraha) from `sanskrit-samas-v1`
Total	~1,000,211	one canonical training file

Design rationale

The splitter's task at a seam is phonetically identical for external sandhi and internal compound boundaries — the model must fire at every valid seam and stay quiet inside single words. Merging (not split-training) lets the shared phonetics reinforce both, while the samas rows teach stem-co-occurrence for the identity-concat class (no phonetic signal) and the corpus rows keep the real distribution. See benchmarks/PLAN_SAMAS.md.

Schema (per row)

Field	Type	Description
`seq`	string	SLP1 surface (input to the model)
`bnd`	list[int]	Boundary positions (last-char indices of each piece except the last)
`n_boundaries`	int	len(bnd)
`source`	string	`sandhi_corpus` or `samas`
`samasa_type`	string (opt)	one of 7 types, for samas rows
`stem1`, `stem2`	string (opt)	constituent stems, for samas rows
`verified`	bool	always true

Samasa type coverage (in the samas portion)

Type	Meaning	n
tatpuruṣa	determinative (X of Y)	62,893
karmadhāraya	appositional (X that is Y)	62,893
dvandva	copulative (X and Y)	59,924
bahuvrīhi	exocentric "having X"	60,172
avyayībhāva	adverbial	7,097
amreḍita	reduplicative	5,091
dvigu	numerical	338

Rebuilding

bash

# samas dataset (deterministic, seed 42)
python3 benchmarks/build_samas_data.py --out data/samas_V1.jsonl

# merged dataset (deterministic, seed 42)
python3 benchmarks/merge_datasets.py \
    --sandhi data/train_V2_full.jsonl \
    --samas data/samas_V1.jsonl \
    --out data/train_V3_sandhi_samas.jsonl