amritha27/cl3410-phase1

CL3410 Phase 1 — Malayalam and Assamese language-model corpora

Two independently built pretraining corpora with their own tokenizers: Malayalam as the higher-resource language and Assamese as the lower-resource one. Nothing is shared between them — separate sources, separate cleaning thresholds, separate vocabularies, separate models. Only the language-agnostic pipeline code is common, parameterised per language.

Everything here was collected and cleaned for this project. No pretrained tokenizer or language model was used at any stage; the quality filters are fitted from scratch in NumPy and the tokenizers are trained from scratch with SentencePiece.

Headline numbers

"Manual" means collected by this project — scraped, OCR'd or transcribed — as opposed to taken from an existing published corpus. The token share exceeds the word share because manually collected text (OCR'd books, literary prose) has higher fertility than crawled text.

Layout

lma-dataset/processed/{ml,as}/ train-.jsonl.gz val-.jsonl.gz test-*.jsonl.gz corpusstats.json headline totals, per-split, per-source stagecounters.json the full filter funnel, per source token_stats.json exact token counts by source type

lma-dataset/{malayalam,assamese}/tokenizer/ {ml,as}sp6000.model SentencePiece model {ml,as}sp6000.vocab vocabulary {ml,as}sp6000.config.json every hyperparameter used tokenizerreport.json fertility, compression, training input token_stats_.json unknown-token and utilisation statistics

lma-dataset/{malayalam,assamese}/manual/ every manually collected source, pre-cleaning, with provenance

report/ phase1malayalam.md phase1assamese.md phase1_tokenizers.md figures/ 56 plots tables/ 12 CSVs

Each document carries its provenance, so any statistic here can be recomputed and the manual/downloaded split re-derived:

docid, text, source, sourcetype (manual|downloaded), url, title, license, datecollected, scriptratio, synthetic, dupcluster, splitkey

Loading

from datasets import load_dataset

ml = load_dataset("amritha27/cl3410-phase1", "malayalam", split="train")
As = load_dataset("amritha27/cl3410-phase1", "assamese", split="train")

The tokenizers are plain SentencePiece models:

import sentencepiece as spm
from huggingface_hub import hf_hub_download

p = hf_hub_download("amritha27/cl3410-phase1",
                    "lma-dataset/malayalam/tokenizer/ml_sp_6000.model",
                    repo_type="dataset")
sp = spm.SentencePieceProcessor(model_file=p)

How it was built

Nine stages, cheapest filters first (the Yi recipe): heuristics, then learned filters, then deduplication.

Calibration. Thresholds come from percentiles of each source's own measured distribution rather than being assumed — a symbol-ratio bar that suits Wikipedia is wrong for a scanned book.

Learned filters, from scratch. A character 5-gram LM with stupid backoff over hashed count tables gives perplexity; a hashed logistic-regression classifier (2^18 features) gives a quality score. Crawled text is ranked within its own distribution, not against the curated distribution — the LM is fitted on curated text, so crawl scores worse under it by construction, and judging one against the other filters register rather than quality.

Manual text is privileged, deliberately. Manual sources are processed first and seed the deduplication hash set, so a collision is resolved in favour of the manual copy. Manual sources also get relaxed symbol and script thresholds, because OCR output genuinely is noisier and holding irreplaceable hand-collected text to a crawl's bar discards it.

Deduplication. Exact (blake2b) and paragraph-level boilerplate removal, then near-duplicates via MinHash (100 permutations, 5-word shingles) with LSH banding at 20 x 5, whose S-curve is steepest at Jaccard 0.549. Clusters are formed with union-find.

Decontamination. Splits are assigned by hashing the near-duplicate cluster id, not the document id, so no near-duplicate can straddle a train/test boundary. This is asserted, and both corpora report zero violations checked exactly.

Tokenizers

SentencePiece unigram, vocabulary 6,000, byte fallback enabled, character_coverage=0.9995, identity normalisation (the pipeline has already normalised), trained on the train split only.

Why 6,000. This is a parameter-budget decision. For a decoder-only transformer with tied embeddings and ffn_mult=4, total parameters are V*d + L*(12d^2 + 2d). Fitting each vocabulary to a 25M budget, the width holds at d_model=480 up to 6,000 and drops to 448 at 8,000. So 6,000 is the largest vocabulary that still affords the full width: anything smaller has an identical body and worse fertility, anything larger trades 2.85M working parameters for 0.14 fertility. Embedding parameters do no computation — they are a lookup — so moving budget into them is moving it out of the part that learns.

A nine-point sweep (2k to 48k) backs this: report/tables/vocab_sweep_*.csv and the fertility_vs_vocab, parameter_budget and capacity_tradeoff figures.

Unknown tokens. With byte fallback, UNK is zero by construction, so that number alone says nothing. The reports also give byte-fallback rate (split into structural whitespace and real content), OOV character rate, and a counterfactual UNK rate measured by re-encoding held-out text with byte fallback disabled — 3.79e-02 for Malayalam, 2.58e-02 for Assamese.

Limitations

• —Assamese is 480.7M train-split tokens, 3.9% under the 500M target (490.7M across the whole corpus, 1.9% under). The downloaded pool across all twelve sources is exhausted after filtering: stage 02 reports 14.3M words of budget unused. This needs new sources, not more of the existing ones. report/phase1_assamese.md documents it.
• —Malayalam's train split is 0.19% under 500M while its whole corpus is 1.9% over. The target is stated over training tokens, so the train figure is the one that counts; the gap is the 2% held out for validation and test, not a data shortage.
• —A small amount of Assamese is machine-translated (Sangraha's synthetic/asm_Beng). It is allocated only against the shortfall real sources cannot fill, is flagged per document with synthetic: true, and its costs are set out in the project repository.
• —~1.9% of Malayalam manual text is Sanskrit in Malayalam script (Wikisource mūlam editions) — 0.45% of the corpus. Malayalam's vocabulary is already heavily tatsama, so these pieces are not foreign.
• —Third-party corpora are not mirrored here. IndicCorp v2, Sangraha, CulturaX, FineWeb-2, Varta and Samanantar are redistributions under their own licences; they are reproducible from the collection code, and their per-source counts before and after cleaning are in report/tables/sources_*.csv.
• —Assamese is written in the Bengali script and is easy to mistake for Bengali. Language identification keys on the Assamese-only letters ৰ (U+09F0) and ৱ (U+09F1) rather than on the Unicode block.

Licensing

Mixed, per source, and recorded per document in the license field. Wikipedia and Wikisource content is CC BY-SA; scanned books are public-domain works; scraped news is used under fair-dealing for non-commercial academic research and is attributed by URL. Anyone reusing this should check the source and license fields for the subset they intend to use rather than treating the collection as uniformly licensed.

Built for CL3410 (Language Models and Agents), IIIT Hyderabad.