Sidharth1743/indicphi

IndicPHI

Synthetic Indian clinical documents with PHI/PII span labels for 23 languages (22 scheduled Indian languages + English). All identifiers are synthetic surrogates, not real patients.

Two Hub configs at the repo root:

Also on the repo: curated.jsonl / curated.parquet (undivided curated set) and figures/.

<!-- figures/overview.png: canvas header + 4 stats + funnel chart -->

[image]

Personas come from `nvidia/Nemotron-Personas-India`. English drafts are generated with Sarvam-105B (NeMo Data Designer). S4b tag-preserving translation uses Sarvam-105B first; Gemma 4 31B (gemma-4-31B-it) ran locally on an RTX 3090 as the fallback after Sarvam failed script or tag checks (rare-script languages: Bodo, Manipuri, Santali, Kashmiri, Sindhi, Dogri, Sanskrit; same ladder can apply to other languages). Grok-4.3 is the linguistic judge. Rows then pass a deterministic auditor (checksums, DICS, script purity) and NeMo Curator dedup.

Load

from datasets import load_dataset

ds = load_dataset("Sidharth1743/indicphi")  # JSONL documents
row = ds["validation"][0]
print(row["text"][:200])
print(row["spans"][:3])

gliner = load_dataset("Sidharth1743/indicphi", "gliner")
print(len(gliner["train"][0]["tokenized_text"]), gliner["train"][0]["ner"][:3])

Record format (default)

Example (English SMS, validation):

HSP-UT-AAA2E4
Naveen Mishra, MRN-7831D797B4, urgent psychiatry screening at Rae Bareli District Hospital for panic attack. APT-240521-001 with Dr. Anil Kumar. Call 6903169442 for details.

GLiNER format (gliner)

Use this config to fine-tune GLiNER. Each row is:

{"tokenized_text": ["MRN-15FE5B1DAF", "HSP-AS-935C09", "টোকা", "..."], "ner": [[0, 0, "MRN"], [1, 1, "HOSPITAL_ID"], ...]}

ner is [start_token, end_token, LABEL] with inclusive token indices.

These files are not a dump of the JSONL tokenized_text field. The JSONL tokenizer was Latin-centric and split Indic graphemes into characters (টোকা → ট,ো,ক,া), so mean length was 589 tokens and 68.3% of train documents exceeded GLiNER’s 384-token window.

The uploaded gliner_*.json files re-tokenize with Unicode letter/mark/number clusters (hyphenated IDs stay one token) and split leftover long docs into overlapping windows on entity boundaries. Span count is unchanged (390,105). Window counts are higher than document counts because of that split (train 22,554 → 22,889; eval 3,982 → 4,054). Mean tokens per window is 189; 0 windows are over 384.

All 50 labels appear across the two GLiNER splits (STUDENT_ID is train-only; CREDIT_CARD_NUMBER and CVV are eval-absent).

Languages

<!-- figures/language-yield.png: horizontal bar chart "Language yield vs quota" -->

[image]

Quota was 1,272 personas per language (Bodo 1,268). Split is stratified by language; all documents from one persona UUID go to train or eval, never both.

Santali, Manipuri, Sanskrit, Tamil, Kashmiri, and Bodo lose more rows at the linguistic judge (script / fluency), not at the split.

<!-- figures/filter-fails.png: S5 and S6 fail charts -->

[image]

Document types and domains

<!-- figures/doc-types.png: horizontal bars of 14 doc types -->

[image]

Types (14): dischargesummary (2090), radiologyreport (2075), ertriagenotes (2063), opdslip (2061), phcregister (2052), telemedicinetranscript (2048), prescription (2031), referralletter (1884), labreport (1854), automatedsms (1752), ashaworkernote (1722), insuranceclaim (1710), hospitalbilling (1614), surgical_note (1580).

Domains in this release: tbncd (4632), oncologychronic (4611), generalmedicine (4596), maternalhealth (4488), psychiatry_behavioral (4265), surgical (3944). Config also defines paediatric (max age 17). This curated set has no paediatric rows — sampled personas are adults (age 18–111).

Entity types (50)

Personal: PATIENT_NAME, RELATIVE_NAME, DOB, AGE, GENDER, RELIGION, CASTE, OCCUPATION

Government IDs: AADHAAR_NUMBER, PAN_NUMBER, PASSPORT_NUMBER, DRIVING_LICENCE, VOTER_ID, BPL_RATION_CARD, TAX_ID

Contact / location: PHONE_NUMBER, TELEPHONE_LANDLINE, EMAIL_ADDRESS, RESIDENTIAL_ADDRESS, DISTRICT, VILLAGE, PIN_CODE, IP_ADDRESS, URL

Financial: BANK_ACCOUNT_NUMBER, IFSC_CODE, BANK_ROUTING_NUMBER, CREDIT_CARD_NUMBER, CVV, PIN

Other: EMPLOYEE_ID, STUDENT_ID, IMEI_NUMBER, MAC_ADDRESS, VEHICLE_REGISTRATION

Healthcare IDs: ABHA_ID, ABHA_ADDRESS, MRN, PATIENT_ID, HOSPITAL_ID, ADMISSION_NUMBER, ENCOUNTER_ID, APPOINTMENT_ID, REFERRAL_ID

Providers / facility: DOCTOR_NAME, ASHA_WORKER_NAME, HOSPITAL_NAME, BED_NUMBER, WARD_NUMBER, INSURANCE_POLICY_NUMBER

Mean ISED is 0.41 because closed-class types (gender, age, religion, IFSC) repeat by design. High-cardinality IDs stay diverse (HOSPITAL_ID 0.997, MRN 0.983, PHONE_NUMBER 0.978).

Intended use

• —default: inspect documents, character spans, language / doc-type metadata.
• —gliner: fine-tune GLiNER (or similar token NER) on the re-tokenized windows.

Do not use as real medical records, for clinical decisions, or as if the identifiers belong to real people.

Limitations

• —Fully synthetic. Clinical facts are generated, not extracted from hospitals.
• —Some Indic documents still contain English names, hospital names, or Latin IDs (allowed for several entity types).
• —Yield is lower for Santali, Manipuri, Sanskrit, Tamil, Kashmiri, and Bodo.
• —No paediatric domain rows in this drop.
• —DICS requires repeated identity tags (phone, name, MRN, …) to share one value; inconsistent translations were dropped.
• —Do not train GLiNER on JSONL tokenized_text; that tokenizer splits Indic scripts. Use the gliner config.
• —GLiNER also spends part of its 384-token budget on label prompts, so max_len at train/inference still matters.

License

MIT. Personas are sampled from NVIDIA Nemotron-Personas-India; respect that dataset’s license as well.

Citation

@misc{indicphi2026,
  title        = {IndicPHI: Synthetic Indic Clinical PHI/PII NER Dataset},
  author       = {HackGrid},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  note         = {Run 20260830T222852, 26536 curated documents, 23 languages}
}

Pipeline: IndicPHI repository (NeMo Data Designer + Curator, Sarvam-105B generation/translation, Gemma 4 31B local RTX 3090 S4b fallback, Grok-4.3 judge).