Modelpublic

pratham-commits/vaani-gujarati-slm

sourceHugging Faceapache-2.0updated 8d agoView on Hugging Face

1likes262downloads

Model Card

Vaani — Gujarati Small Language Model (~110M)

Submission: IAIRO / PRAMANA SLM++ · Domain: Medical QA (Gujarati) · Params: 109.5M (≤1B) · Base: trained from scratch

Vaani is a 110M-parameter Gujarati language model pretrained from scratch on a Gujarati corpus, then instruction-tuned and adapted to the medical-QA domain. It is built for Gujarati instruction following and grounded (open-book) medical question answering.

1. Model & Architecture

Field	Value
Model name / version	Vaani (med-v5, final)
Base model	From scratch
Parameter count	109.5M
Layers	12
Model dim (d_model)	768
Attention	Multi-Head Attention (MHA); nhead = nkvhead = 12, headdim 64
Normalization	RMSNorm (pre-norm)
Activation / FFN	SwiGLU, hidden mult 8/3 (~2048)
Positional encoding	RoPE (theta 10,000)
Context length	2,048
Embeddings	Tied input/output embeddings
Tokenizer	Character-level BPE, vocab 32,000, trained for Gujarati (1.30 tokens/word)
Precision	bf16

2. Domain & Task

—Domain: Medical (Gujarati).
—Primary task: Multiple-choice medical QA + Gujarati instruction following.
—End user / use case: Gujarati-language assistant for grounded medical reading comprehension (answering from provided context), not a source of unaided medical facts.

3. Data

Field	Value
Pretraining corpus	Gujarati corpus, ~6.78B tokens (Sangraha + Wikipedia-gu + FineWeb-2; see §8)
Pretraining tokens seen	~13.63B (13,000 steps × 1.05M tokens/step, ≈2 epochs)
Instruction SFT	sftv4.jsonl (~150k): WikiChat + Dolly + WikiHow, plus format skills (JSON / extraction / transformation / exact-count lists); see §8
Medical SFT	medicalsftv3.jsonl (~114k): MedMCQA-gu closed-book + raw-grounded + redacted-grounded + 30% general mix
Contamination check	domain eval uses a held-out MedMCQA-gu validation split, disjoint from SFT

4. Training

Field	Value
Objective	Causal LM (next-token)
Effective batch	512 sequences (micro-batch 8 × grad-accum 64) × 2,048 ctx = 1.05M tokens/step
Steps	13,000 (pretraining)
Optimizer	AdamW (betas 0.9/0.95, eps 1e-8, fused, decoupled weight decay 0.1)
LR schedule	WSD: warmup 700 → peak 4e-4 → cosine decay to 4e-5 (final 20%)
Hardware	1× NVIDIA L4 (24 GB)
GPU-hours (final model path)	~100 h (pretraining ~96 + instruction SFT v4 ~3 + medical SFT v5 ~1)
GPU-hours (incl. SFT ablations)	~106 h

5. Post-Training (SFT)

Field	Value
SFT performed	Yes
Method	Full fine-tune (not LoRA)
Instruction SFT	3 epochs, effective batch 32
Medical SFT	2 epochs, effective batch 32, on the instruction-tuned checkpoint
Chat template	`વપરાશકર્તા:\n{question}\nસહાયક:\n{answer}`

6. Evaluation (self-reported internal eval)

Level 1 — Training health: PASS (all gates) — loss reduction 68.2% (10.53→3.35), convergence, no NaN/Inf, final val–train gap 2.0%.

6.1 Development progression (Vaani checkpoints)

MCQ = acc_norm, fixed n=80 Gujarati dev set. Instruction = rule-based checkers (9 buckets). Open-book = MedMCQA-gu val, closed vs with-context (raw-grounded).

Stage	MCQ (n=80)	Instruction	Open-book (closed → w/ctx)
Instruction SFT v1	36.3%	35.9%	-
Instruction SFT v2	36.3%	46.0%	-
Instruction SFT v3 (targeted aug)	31.3%	68.7%	-
Format SFT v4 (JSON/extract/lists)	28.8%	66.7% (71.2% greedy)	-
Medical SFT v3 (raw-grounded)	33.8%	68.2%	26.6% → 30.8%
Medical SFT v4	41.3%	66.7%	28.6% → 38.2%
Medical SFT v5 (final)	41.3%	69.7%	28.8% → 44.1%
Medical SFT v6 (redacted)	-	-	28.7% → 42.8%

Instruction-following rose 36% → 70% via targeted augmentation and format skills. Open-book (grounded) accuracy rose monotonically 30.8% → 38.2% → 44.1% across medical stages, lift growing +4.2 → +9.6 → +15.3. v6 (redacted-heavy) regressed vs v5, so v5 is submitted. On the larger cross-model sets, v5 scores MCQ 38.6% (n=145) and open-book with-context 44.1% (n=1,858).

6.2 Comparison vs external models

Same harness for all. MCQ = 145-item Gujarati set (acc_norm, log-likelihood); Instruction = rule-based checkers; Open-book = MedMCQA-gu val, with vs without context (Vaani n=1,858; external models n=500).

Model	Params	MCQ	Instruction	Open-book w/ctx	Grounding lift
Vaani (final)	110M	38.6%	70%	44.1%	+15.3
gemma-2-2b-it	2B	37.2%	67%	37.8%	+9.6
sarvam-1 (base)	2B	36.6%	24%	40.2%	+11.4
Navarasa-2.0	2B	42.8%	54%	35.0%	+10.0
Qwen2.5-7B-Instruct	7B	51.0%	53%	44.0%	+18.0

Takeaways:

—Instruction-following leads the field: Vaani 70% > gemma-2B 67%, Navarasa-2B 54%, Qwen-7B 53%. Even Qwen-7B fails exact-count lists (list_3/5 0%, numbered 14%) — skills Vaani was explicitly trained on.
—Open-book comprehension is best in test: Vaani 44.1% tops Qwen-7B (44.0%) and all 2B models. A 110M model leads on grounded reading.
—General MCQ: Vaani 38.6% beats gemma-2B (37.2%) and sarvam-2B (36.6%); trails Navarasa-2B (42.8%) and Qwen-7B (51%). Closed-book recall scales with size, as expected. Vaani is built to use provided context, not recall facts unaided.

7. Reproducibility

—Weights: https://huggingface.co/pratham-commits/vaani-gujarati-slm
—Datasets: https://huggingface.co/datasets/pratham-commits/vaani-gujarati-sft-data
—Inference: eval.py (research harness) / chat.py (interactive); greedy for structured/objective tasks (temp 0), sampling for open-ended.
—Dependencies: torch 2.9.1+cu129, transformers 4.57.6, tokenizers 0.22.2, huggingface_hub<1.0 (Python 3.10)
—Known limitations / failure modes:
—Closed-book medical factual recall is limited by 110M capacity; strongest when answering from provided context (open-book/RAG).
—Structured JSON output was learned only for the fixed schemas seen in training (e.g. name/age/city) and does not generalize to arbitrary keys — not exposed as a general capability. Exact-count lists require greedy decoding; sampling with no-repeat-ngram degrades them.
—Gujarati-only; not tuned for code-switching or other Indic languages.

8. Data Sources, Licenses & Attribution

Pretraining corpus (described, not re-hosted):

—AI4Bharat Sangraha (verified / unverified / synthetic, gu) — CC BY-4.0. Khan et al., 2024 (arXiv:2403.06350).
—Wikipedia (gu, 20231101) — CC BY-SA 4.0. Wikimedia Foundation.
—FineWeb-2 (guj_Gujr) — ODC-BY. Penedo et al., 2024 (arXiv:2406.17557).

Instruction SFT sources (in sft_v4.jsonl):

—Dolly (databricks-dolly-15k, translated to Gujarati) — CC BY-SA 3.0. Databricks.
—WikiHow (Gujarati) — derived from WikiHow content (CC BY-NC-SA); see license note below.
—Wiki_Chat — Wikipedia-derived instruction/QA (CC BY-SA).
—Anudesh (AI4Bharat IndicLLMSuite) was evaluated and dropped for quality; not included.

Medical data:

—MedMCQA — Pal, Umapathi & Sankarasubbu, 2022 (arXiv:2203.14371). MedMCQA-gu is a Gujarati translation of MedMCQA items, decontaminated against the held-out validation split.

External comparison models (benchmarked only, not redistributed): google/gemma-2-2b-it (Gemma Terms of Use), sarvamai/sarvam-1, Telugu-LLM-Labs/Indic-gemma-7b-finetuned-sft-Navarasa-2.0 (Gemma Terms of Use), Qwen/Qwen2.5-7B-Instruct (Apache-2.0).

License note: Vaani's weights are released under Apache-2.0. The SFT data combines sources under different licenses (CC BY-SA, and WikiHow-derived content under a non-commercial share-alike license). See the dataset card for per-source licensing; downstream users should respect each source's terms.

9. Citations

—Khan et al. (2024), IndicLLMSuite / Sangraha, arXiv:2403.06350
—Penedo et al. (2024), FineWeb / FineWeb-2, arXiv:2406.17557
—Pal, Umapathi & Sankarasubbu (2022), MedMCQA, arXiv:2203.14371
—Hoffmann et al. (2022), Chinchilla, arXiv:2203.15556; Sardana et al. (2024), arXiv:2401.00448
—Full design-decision references: see DECISIONS.md in the project repo.

Acknowledgements

Built during the IAIRO / PRAMANA SLM++ bootcamp.