CodeIsAbstract/sanskrit-sandhi-boundaries-v2

Sanskrit Sandhi Boundary Dataset (V3 — verified, category-complete)

Training data for the sandhi boundary-detection model in `CodeIsAbstract/sanskrit-sandhi-boundary-v2`. The task: given a sandhi-joined string (a compound or multi-word string), predict the character positions where independent words end, so a downstream Sanskrit tokenizer can split it into complete, independent tokens.

This is the verified release: every row has been passed through a deterministic sanitizer (clean_train_data.py) and a full reliability audit (verify_dataset.py). All boundary labels follow the exact convention of the training data generator (fix_train_labels.fixed_join).

Files

Use `data/train_V2_full.jsonl` for a single-run train. It contains all base corpus data PLUS the hand-curated category-completeness rows (s→s doubling, o→v, avagraha, kRSNo/RSNa prefixes, tacca-type), so you do not need to concatenate files or run a separate fine-tune step.

Row format

{"seq": "yepanTAnaitiparItyopadaDIta",
 "bnd": [1, 8, 11, 18],
 "ctx": [[1, 0], [1, 0], [0, 0], [0, 0]],
 "it": [[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0], ...],
 "n_boundaries": 4,
 "sentence_id": 12345,
 "verified": true}

• —seq — the sandhi-joined string (SLP1 transliteration; may contain literal spaces where the corpus itself had them, e.g. avagraha rows).
• —bnd — 0-based last-character index of each word boundary (character i of seq is the last char of word k). Example: "rAmo"|"gacCati" → seq="rAmogacCati", bnd=[3] because o at index 3 ends the first word.
• —ctx — per-boundary context flags (used by the decoder).
• —it — per-boundary token indices (used by the decoder).
• —verified — set during the label-fix pass.
• —n_boundaries — len(bnd).

Boundary convention (important)

The convention is the exact one emitted by the training-data generator (fix_train_labels.fixed_join, rules from the vidyut data rules.csv):

• —bnd = len(merged_w1) - 1 where merged_w1 is the surface form of word 1 after sandhi application.
• —The final word of a row is always the remainder after the last boundary — a boundary is never placed at len(seq)-1 (string end). The base dataset (691,675 rows) contains zero boundaries at len-1; this invariant is enforced by the sanitizer and the audit.
• —Avagraha rows follow "o ' " → 'o' in word 1, "'" in word 2 (rAmo'vadat → bnd=[3], pieces rAmo + 'vadat).
• —Shared-vowel boundaries keep the merged vowel in word 1 (a+i→e → rAmeti = rAme + ti, bnd=[3]).

Data quality — what was verified

Sanitization (`clean_train_data.py`) — base input was 697,004 rows; 5,329 malformed rows were dropped (never silently fixed):

Reliability audit (`verify_dataset.py`) — for every row:

1. 1.seq is a non-empty string
2. 2.bnd strictly increasing, all in [0, len(seq)-2] (final word = remainder)
3. 3.no duplicate boundaries
4. 4.every piece is non-empty and ''.join(pieces) == seq exactly

Current status (both files): 0 invalid rows, exit code 0.

All 63 hand-curated rows (benchmark sentences, visarga/avagraha surfaces, category-completeness additions) were additionally verified against the vidyut sandhi engine: every boundary is splitter-expressible (Splitter.split_at probe at {bnd-1, bnd, bnd+1}) and passes the two-word rejoin check (_two_word_rejoins).

Category completeness

The targeted set was built to close every measured representation gap in the base data (a neural boundary model's output probability is a function of how well a junction category is represented — see the model card). Junction counts in data/train_V2_targeted.jsonl (pair = (seq[bnd], seq[bnd+1])):

\ `t→c` is 0 by design*: tat+ca surfaces as tacca with the boundary at the doubled consonant (tac + ca → junction c→c), so tacca-type rows are counted under c→c.

The kRSNo / RSNa prefixes (zero examples in the base corpus) are now represented by 7 hand-curated rows (kRSNogacCati, RSNogacCati, kRSNastatra, RSNaSca, kRSNaAgacCati, …).

Model training

# base training
python train_split_model_cuda.py --data data/train_V2_fixed.jsonl \
  --device auto --delay 2 --num-modes 32 --epochs 5 \
  --out checkpoints/delayed_model_save_fixed

# targeted fine-tune (resume from base checkpoint)
python train_split_model_cuda.py --data data/train_V2_targeted.jsonl \
  --resume checkpoints/delayed_model_save_fixed_delay2_v2 \
  --out checkpoints/targeted_ft --epochs 3 \
  --latent-dim 256 --num-modes 64 --layer-types linear,softmax \
  --pos-weight 4.0 --delay 2

Checkpoint format is model.safetensors + config.json, loadable by decode_best.py / decode_v2.py with no changes.

Verification (reproduce)

python verify_dataset.py data/train_V2_fixed.jsonl     # exit 0 = all valid
python verify_dataset.py data/train_V2_targeted.jsonl  # exit 0 = all valid
python clean_train_data.py --in <raw.jsonl> --out <clean.jsonl>

License note

License intentionally left as other: the corpus provenance (source texts of the raw sandhi pairs) has not been fully verified for redistribution. The labels and build/verification tooling in this repo are released for use with the model training pipeline; if you redistribute the raw seq strings for purposes beyond model training, verify the underlying text provenance first.