subidhkhanal/sarvam-1-hindi-citizen-profile-lora

Sarvam-1 · Hindi → Citizen Profile JSON (LoRA)

A QLoRA fine-tune of `sarvamai/sarvam-1` that converts spoken-Hindi self-descriptions into a strict 20-field CitizenProfile JSON schema. Designed for form-filling assistants at government service centers, banking KYC, and welfare-scheme applications, where applicants describe themselves in natural Hindi and the downstream form needs structured fields.

Headline (290-entry held-out test set):

Source code, data preparation pipeline, and training/eval scripts: [github.com/subidhkhanal/hindi-form-agent](https://github.com/subidhkhanal/hindi-form-agent).

Quick start

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

BASE = "sarvamai/sarvam-1"
ADAPTER = "subidhkhanal/sarvam-1-hindi-citizen-profile-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True,
    ),
    device_map="auto",
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

# Exact prompt format used at training time
INPUT_DELIMITER = "हिंदी पाठ:"
OUTPUT_DELIMITER = "संरचित JSON:"

hindi_text = (
    "मेरा नाम रामलाल है। मेरी उम्र चालीस साल है। "
    "मैं बिहार के मधुबनी जिले में रहता हूं। खेत मजदूरी का काम करता हूं, "
    "महीने में लगभग पच्चीस हजार रुपये कमाता हूं। आधार कार्ड और बैंक खाता है।"
)
prompt = f"{INPUT_DELIMITER}\n{hindi_text}\n\n{OUTPUT_DELIMITER}\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=512, do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
gen = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# Model may emit trailing junk after the JSON; raw_decode handles it
predicted, _ = json.JSONDecoder().raw_decode(gen)
print(json.dumps(predicted, ensure_ascii=False, indent=2))

Expected output:

{
  "full_name": "रामलाल",
  "age": 40,
  "gender": "male",
  "marital_status": null,
  "district": "मधुबनी",
  "state": "बिहार",
  "occupation": "खेत मजदूर",
  "monthly_income_inr": 25000,
  "has_aadhaar": true,
  "has_bank_account": true,
  "bank_name": null
}

(plus null for the remaining fields)

Output schema

20-field Pydantic CitizenProfile (full definition in repo). All fields are Optional — the model is trained to emit null for fields the input does not mention, rather than hallucinate.

Closed-set (Literal) fields use canonical labels:

• —gender ∈ {"male", "female", "other"}
• —marital_status ∈ {"single", "married", "widowed", "divorced"}
• —caste_category ∈ {"general", "obc", "sc", "st"}
• —has_aadhaar, has_pan, has_voter_id, has_ration_card, has_bank_account ∈ {true, false}

Government ID fields are booleans only — actual ID numbers are never captured in training data (privacy by design).

Training

Training data

2,270 train / 290 val / 290 test entries from four sources:

Full data prep methodology and the persona/domain references live in `data_prep/` in the GitHub repo.

Evaluation

Strict exact-match field accuracy on a held-out test set (290 entries, never seen during training). Test set is itself produced by a deterministic random.seed(42) split, fully reproducible.

By source

By field (gold-non-null accuracy, sorted)

Highlights:

• —Every closed-set (Literal-typed) field at 100% gold-NN — the model canonicalizes free Hindi text to enum values reliably (अनुसूचित जाति → sc, मुस्लिम → इस्लाम, विधवा → widowed).
• —`bank_name` at 100% (25/25) — the conditional rule (extract iff the speaker named the bank, else null) generalized perfectly.
• —Devanagari number → int: चौवालीस → 44, इकहत्तर → 71, बावन → 52, etc.

Weak spots (district, full_name, occupation) are bottlenecked by:

1. 1.HiNER label noise (party names mistagged as PER, locations truncated).
2. 2.Strict exact-match: predicting "स्टेशनरी दुकानदार" when gold is "दुकानदार" counts as wrong even though both are correct. Real semantic accuracy is higher.

Intended use

Demo / research / educational. Drop-in inference component for Hindi form-filling assistants, voice-to-form pipelines, and structured-extraction research.

Not for production use without further validation. Government ID-related fields are booleans only (never extracts ID numbers — that's a deliberate privacy boundary, not an oversight).

Limitations

1. 1.Strict exact-match underreports. Predictions semantically equivalent to gold (canonical spelling variants, more-specific occupations) score as wrong. A normalized or semantic-match metric would raise the field-accuracy number meaningfully without changing the underlying capability.
2. 2.HiNER label noise. The 2,000 HiNER training entries provide real-Hindi distributional signal but carry inherited labeling quirks (party names as full_name, multi-word locations truncated). The data-prep stoplist filters the worst cases; some residual noise reaches the model.
3. 3.Generation may over-run past the JSON. The model occasionally emits duplicate JSON blocks or Hindi narrative after the closing }. Always parse with json.JSONDecoder().raw_decode() (takes the first valid object) or use the StoppingCriteria provided in the repo's evaluate.py.
4. 4.First-person dense inputs are the sweet spot. Sparse third-person news sentences (HiNER-style inputs) still produce well-formed JSON but value accuracy drops — that's a property of HiNER labels, not a model failure.

License

The adapter inherits the base model's license: Sarvam non-commercial license. Not for commercial use.

Citation / acknowledgement

Built on top of Sarvam-1 by Sarvam AI, and the HiNER-collapsed NER dataset by AI4Bharat / CFILT. See the GitHub repo for the full data prep acknowledgements and per-source methodology.

Full source, data, and reproducible training scripts: [github.com/subidhkhanal/hindi-form-agent](https://github.com/subidhkhanal/hindi-form-agent).