santoshgairesharma/nepmedjp-mt5-xl

NepMedJP mT5-XL (Continued Pre-trained)

Model ID: `santoshgairesharma/nepmedjp-mt5-xl` Base model: `google/mt5-xl` (~3.7B parameters) Stage: Continued pre-training (T5-style span corruption) on English medical, Japanese, and Nepali text Project: NepMedJP

This is a general-purpose continued-pretrained language model, not a task model. It has only ever been trained on the span-corruption objective. It is not a summarization model, not a translation model, and not instruction-tuned. It will not respond sensibly to task prompts such as summarize: or translate: — those prefixes were never in its training data. To use it for any task, you must fine-tune it first.

This checkpoint is a continued-pretrained mT5-XL whose domain and language coverage has been adapted to English medical, Nepali (medical and general), and Japanese (general and medical) text. The adaptation is measured purely as improved span-corruption loss/perplexity in those domains (see Evaluation) — that is, better language and domain modelling, not task capability.

Within the NepMedJP project this checkpoint serves as the Stage A backbone that later stages fine-tune for Japanese→Nepali medical summarization. That downstream use is one application of this model, not a description of it.

Model summary

Intended use

Direct intended uses

This is a base checkpoint. Every intended use involves training something on top of it:

• —Initialization for fine-tuning on any EN/JA/NE biomedical downstream task (LoRA or full) — summarization, translation, classification, QA, etc. Japanese→Nepali medical summarization is the task NepMedJP itself pursues, but nothing about this checkpoint is specific to it.
• —Further continued pre-training / domain adaptation in EN/JA/NE biomedical text.
• —Research on cross-lingual medical transfer involving Nepali (low-resource) and Japanese — e.g. probing, representation analysis, or perplexity-based domain studies.

Out-of-scope / not intended

• —Not a summarization model. No summarization data or task prefix was used in training. Summarization requires a separate fine-tuning stage.
• —Not usable zero-shot for any task. It has no instruction-following, chat, or task-prefix behaviour; prompting it directly yields span-corruption-style continuations, not answers.
• —Not a clinical decision-support system. Do not use outputs for diagnosis, treatment, or patient care without qualified clinical review.
• —Not a replacement for professional translation or medical authorship.

How to use

1. As a base to fine-tune (the intended path)

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "santoshgairesharma/nepmedjp-mt5-xl"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)

# Attach your task head / LoRA adapter and train on your own labelled data.
# Nothing below this line works until that fine-tuning has happened.

2. Sanity-checking the pretrained objective (span infilling)

The only thing this checkpoint does out of the box is fill in masked spans, because that is the sole objective it was trained on. Masked spans are marked with sentinel tokens (<extra_id_0>, <extra_id_1>, …) and the model predicts the missing text:

text = "मधुमेह एक <extra_id_0> रोग हो जसमा रगतमा <extra_id_1> मात्रा बढ्छ।"

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
# -> sentinel-delimited span predictions, e.g. "<extra_id_0> ... <extra_id_1> ..."

This is a diagnostic, not an application. Useful output for a real task requires fine-tuning.

Note: This release is the continued-pretrained base only. Japanese→Nepali summarization capability lives in the separate NepMedJP LoRA / fine-tuned stage (Stage C) and is not present in these weights.

Training data & sampling strategy

Continued pre-training interleaves five corpora with domain-priority sampling (inspired by mT5 temperature sampling \(p(L) \propto |L|^\alpha\), but here using fixed task-informed weights). English medical and project-constructed medical Nepali are heavily prioritized for cross-lingual medical transfer; general JA/NE corpora support language competence.

Caps (to control steps / cache size): English medical ≤ 3,000,000 records; Japanese general ≤ 2,000,000 records. Split: 95% train / 5% validation per source (seed 42), then interleave_datasets with the weights above (stopping_strategy="all_exhausted" on train).

Full narrative strategy, design rationale, and hyperparameter table: see `training/PRETRAINING_STRATEGY.md`. Exact training code used for this run: `training/pretraining_script.py`.

Training procedure

1. 1.Load google/mt5-xl with dropout_rate=0.1; enable gradient checkpointing.
2. 2.Tokenize texts (no truncation), concatenate, and chunk to expanded input length for span corruption at max length 1024.
3. 3.Apply T5 MLM collator: noise density 0.15, mean noise span length 3.0.
4. 4.Train with Hugging Face Trainer: Adafactor, bf16, LR 1e-4, warmup 1,000, eval/save every 10,000 steps, load_best_model_at_end on eval_loss.
5. 5.Run finished at step 77,075; final weights saved under final_model/.

Wall-clock (from metrics log): ~35,000+ minutes of logged training/eval time on the original run hardware (A100-class recommended).

Evaluation

Baseline vs continued-pretrained (per language / domain)

Span-corruption eval loss and perplexity on held-out slices of each corpus. Baseline = unmodified `google/mt5-xl`. NepMedJP final = this release after 77,075 continued-pretraining steps under the domain-priority configuration.

Machine-readable table: `training/baseline_vs_final_per_language.csv`. Final-only numbers: `training/per_language_eval.csv`.

Takeaways

• —Continued pre-training improves every domain vs stock mT5-XL under the same span-corruption eval.
• —Largest absolute PPL gains: Japanese general (−12.1) and Japanese medical (−9.1); largest relative medical-target gain: Nepali medical (5.20 → 2.57).
• —English medical also improves while remaining a strong medical anchor (3.41 → 2.66).

Configuration note (why this recipe)

A pilot continued-pretraining run using the default mT5 paper-style configuration was evaluated at step 10k against the same baseline. That pilot did not improve—per-domain loss/PPL degraded sharply vs google/mt5-xl (e.g. English medical PPL 3.41 → ~1313). We therefore discarded that configuration and retained the NepMedJP recipe documented here (Adafactor, domain-priority sampling, bf16, effective batch 32, etc.).

Evidence figure from that comparison: `training/plots/baseline_vs_default_mt5_config_step10k.png`

Mixture validation loss (during the successful run)

Eval every 10k steps on the interleaved validation mixture:

Full step-level train/eval curves: `training/metrics_log.csv` and `training/plots/`.

Training plots

• —Final dashboard: `training/plots/metrics_final.png`
• —Intermediate snapshots: metrics_step_0010000.png … metrics_step_0077075.png
• —Baseline vs failed default-config pilot: baseline_vs_default_mt5_config_step10k.png ---

Repository contents

├── config.json / generation_config.json
├── model-00001-of-00003.safetensors
├── model-00002-of-00003.safetensors
├── model-00003-of-00003.safetensors
├── model.safetensors.index.json
├── tokenizer files (spiece.model, tokenizer.json, …)
├── README.md                          ← this model card
└── training/
    ├── PRETRAINING_STRATEGY.md              ← detailed method write-up
    ├── pretraining_script.py                ← exact training script used
    ├── metrics_log.csv                      ← step-level train/eval metrics
    ├── per_language_eval.csv                ← final per-domain eval
    ├── baseline_vs_final_per_language.csv   ← base mT5-XL vs this release
    └── plots/                               ← metric dashboards + baseline figure

Intermediate optimizer checkpoints (checkpoint-60000, etc.) are not included in this Hub release; only the final loadable model and full training evidence packages are published.

Limitations and risks

• —Medical safety: Outputs can be incomplete, outdated, or incorrect. Not validated as a medical device.
• —Data provenance: Silver / translated / curated corpora may contain noise, register mismatch, or domain skew.
• —Language imbalance: Sampling deliberately upsamples medical Nepali and English medical; Japanese perplexity is higher.
• —No task performance is claimed. All evaluation here is span-corruption loss/perplexity, which measures language and domain modelling only. It does not establish summarization, translation, or any other task quality. In NepMedJP those depend on Stage B silver data, Stage C fine-tuning, and Stage D gold physician evaluation — none of which are part of this checkpoint.
• —Bias: Biomedical corpora inherit geographic, institutional, and language biases from source literature.

Ethical considerations

NepMedJP targets improved access to medical information for Nepali speakers interacting with Japanese-sourced materials. Release of open weights and training artifacts is intended to enable reproducible research and community scrutiny. Clinical deployment requires separate validation, governance, and human oversight.

Citation

If you use this model, please cite the NepMedJP project and the original mT5 work:

@misc{nepmedjp-mt5-xl,
  title        = {NepMedJP mT5-XL: Continued Pre-training for Japanese--Nepali Medical NLP},
  author       = {Sharma, Santosh Gaire},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/santoshgairesharma/nepmedjp-mt5-xl}},
  note         = {Continued pre-training of google/mt5-xl}
}

@article{xue2021mt5,
  title   = {{mT5}: A Massively Multilingual Pre-trained Text-to-Text Transformer},
  author  = {Xue, Linting and Constant, Noah and Roberts, Adam and Kale, Mihir and Al-Rfou, Rami and Siddhant, Aditya and Barua, Aditya and Raffel, Colin},
  journal = {NAACL},
  year    = {2021}
}

Codebase: github.com/Santey624/NepMedJPSum

Acknowledgements

Built on `google/mt5-xl` (Apache-2.0). Training used Hugging Face Transformers / Datasets and domain-priority interleaving over EN/JA/NE medical and general corpora constructed or curated for NepMedJP.