buddhist-nlp/mitra-qwen35-base-stage1

mitra-qwen35-base-stage1

The stage-1 continued-pretraining checkpoint of the Dharmamitra model family: Qwen3.5-9B-Base after ~30B tokens of continued pretraining on classical Buddhist corpora (28,500 steps × ~1.05M tokens, 8k context; training loss 1.85 → 1.16).

This is a base (non-instruction) model, released as a research artifact and as the recommended starting point for your own SFT on Buddhist-domain tasks. For an instruction-following model from this lineage use `buddhist-nlp/mitra-qwen35-base-stage2` (this checkpoint + stage-2 SFT); for retrieval use `buddhist-nlp/mitra-qwen35-embedder`.

Corpus and script conventions

The pretraining corpus covers Sanskrit, Tibetan, Buddhist Chinese, and Pāli source texts with related secondary literature.

• —Tibetan is native Tibetan script (Unicode) — unlike some downstream finetunes in this family, stage 1 was not trained on Wylie transliteration.
• —Sanskrit and Pāli are predominantly IAST romanization; Chinese is Chinese script.

Usage

Plain causal-LM completion (no chat template — this is a base model):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("buddhist-nlp/mitra-qwen35-base-stage1")
model = AutoModelForCausalLM.from_pretrained(
    "buddhist-nlp/mitra-qwen35-base-stage1", dtype=torch.bfloat16, device_map="cuda"
)

prompt = "evaṃ mayā śrutam ekasmin samaye bhagavān"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Model details

• —Base: Qwen/Qwen3.5-9B-Base
• —Continued pretraining: 28,500 optimizer steps at ~1.05M tokens/update (~29.9B tokens), 8,192-token packed sequences, bf16, ZeRO-2
• —This checkpoint is the deliberate end of stage 1 (training was moved to stage-2 SFT from here)

Citation

If you use this model, please cite the Dharmamitra project (https://dharmamitra.org). A technical report is in preparation.