FormosanBank/nllb200-formosan-zh-spm8k

nllb200-formosan-zh-spm8k

Direction: Formosan to Traditional Chinese<br> Base model: `facebook/nllb-200-distilled-600M`<br> Recipe: nllb200-spm8k-directional-v9<br> Release: 20260828-121226, validation-selected step 230,000

This is a directional model for 15 Formosan languages. It uses the private_no_bible leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, and explicit direction, source-language, and dialect controls. It does not infer or encode domains from repository or path names. Training uses language-temperature sampling with alpha 0.5; sentence rows have weight 1.0 and explicit lexical rows, when present, have weight 0.25. Dialect tags use 25% dropout so default is a normal inference condition. Synthetic pivot rows are train-only. The model weights are public. The training corpus is distributed separately to authorized FormosanBank members through the access-controlled `FormosanBank/formosan-mt-private` dataset and is not included with the weights.

Model details

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer
from formosan_mt_inference import normalize_formosan


model_id = "FormosanBank/nllb200-formosan-zh-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")
NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dru_Latn', 'pwn': 'pwn_Latn', 'pyu': 'pyu_Latn', 'ssf': 'ssf_Latn', 'sxr': 'sxr_Latn', 'szy': 'szy_Latn', 'tao': 'tao_Latn', 'tay': 'tay_Latn', 'trv': 'trv_Latn', 'tsu': 'tsu_Latn', 'xnb': 'xnb_Latn', 'xsy': 'xsy_Latn'}

def translate(text, lang_code, dialect="default"):
    text = normalize_formosan(text, lang_code)
    tokenizer.src_lang = NLLB_LIDS[lang_code]
    dialect_tag = f"<dialect_{dialect}>"
    if tokenizer.convert_tokens_to_ids(dialect_tag) == tokenizer.unk_token_id:
        dialect_tag = "<dialect_default>"
    prompt = f"<to_zh> <src_{lang_code}> {dialect_tag} {text}"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        decoder_start_token_id=tokenizer.eos_token_id,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids('zho_Hant'),
        max_new_tokens=256,
        num_beams=4,
    )
    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

print(translate("Pa'araw cingra.", "ami"))

The direction, source-language, and dialect tags are part of the training contract. Use default when dialect metadata is unavailable. This model does not use domain or repository tags.

Evaluation

The best checkpoint was selected on validation chrF2. Test and validation contain only eligible, human-translated sentence pairs. Synthetic pivots and lexical entries are train-only. The headline result uses default metadata controls (the default dialect only), so it does not assume access to test-set dialect labels or any inferred domain metadata.

Test empty-output rate: 0.0039%.

Bootstrap confidence intervals were not requested for this release.

Metric signatures:

• —BLEU: nrefs:1|case:mixed|eff:yes|tok:zh|smooth:exp|version:2.5.1
• —chrF2: nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1
• —TER: nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:yes|version:2.5.1

The corpus gate applies 85% train / 5% validate / 10% test to deduplicated human pairs within each language and source where capacity permits. Synthetic pivots are appended to training after that split. Evaluation is sentence-only and contains no lexical or synthetic rows. Standard-tier Formosan text is normalized with formosan-mt-standard-v3. The release requires zero exact, skeleton, one-edit, or configured high character n-gram train/evaluation conflicts. Document overlap is diagnostic. This release passed all leakage gates: exact 0, skeleton 0, one-edit 0, character n-gram

1. 1.Document overlap: 1433.

See eval/metrics.json for sacreBLEU signatures, per-language, source-corpus, dialect, and length diagnostics. publication.json records the corpus, profile, run, and checkpoint hashes used for this release.

Intended use

This model supports research, corpus development, and assisted translation for the 15 included Formosan languages. It is designed for the exact prompt and generation contract shown above.

Limitations

Outputs require knowledgeable speaker review. Aggregate metrics hide large differences among languages and source collections. This model is not suitable for authoritative, medical, legal, or safety-critical translation.