pagantibet/normalisationS2S-nontokenised

1---2license: cc-by-nc-sa-4.03language:4- bo5tags:6- seq2seq7- normalisation8- classical-tibetan9- historical-text10- encoder-decoder11- transformer12- low-resource13- digital-humanities14pipeline_tag: text-generation15---16 17# Tibetan Normalisation - S2S Model (Non-Tokenised)18 19A character-level sequence-to-sequence (S2) encoder-decoder transformer model for the normalisation of Old/Classical Tibetan, converting diplomatic (non-standard, abbreviated) Tibetan manuscript text into Standard Classical Tibetan. This is the **non-tokenised** variant of the model — input and output are raw Tibetan Unicode character sequences without prior word segmentation.20 21This model is part of the [PaganTibet](https://www.pagantibet.com/) project and accompanies the paper:22 23Meelen, M. & Griffiths, R.M. (2026) 'Historical Tibetan Normalisation: rule-based vs neural & n-gram LM methods for extremely low-resource languages' in *Proceedings of the AI4CHIEF conference*, Springer.24 25Please cite the paper and the [code repository](https://github.com/pagantibet/normalisation) when using this model.26 27---28 29## Model Overview30 31Old/Classical Tibetan manuscripts present major normalisation challenges: extensive abbreviations, non-standard orthography, scribal variation, and a near-complete absence of gold-standard parallel data. This model addresses these challenges using a hybrid approach combining a neural sequence-to-sequence transformer with optional rule-based pre-/post-processing and KenLM n-gram language model ranking (the latter applied at inference time; see the [Inference scripts](https://github.com/pagantibet/normalisation/tree/main/Inference)).32 33The model operates at the character level on non-tokenised input, meaning it processes raw Tibetan Unicode strings (syllables and punctuation) without word segmentation. Results from Meelen & Griffiths (2026) show that, for non-tokenised text, the neural model alone performs reasonably well on standard Buddhist texts, while more challenging diplomatic corpora benefit from the addition of rule-based processing and n-gram ranking.34 35### Architecture36 37- **Type**: Character-level encoder-decoder transformer (Seq2Seq)38- **Layers**: 439- **Attention heads**: 840- **Optimiser**: Adam (lr = 0.0005, β1 = 0.9, β2 = 0.997)41- **Label smoothing**: 0.142- **Framework**: PyTorch43- **Training hardware**: RTX ADA 6000 GPU (~5–6 hours training time)44 45Full hyperparameter settings are reported in the Appendix of Meelen & Griffiths (2026).46 47---48 49## Training Data50 51The model was trained on the dataset [`pagantibet/normalisation-S2S-training`](https://huggingface.co/datasets/pagantibet/normalisation-S2S-training) (~2 million rows), which combines:52 531. **Gold-standard data**: 7,421 manually normalised parallel sentence pairs from the PaganTibet corpus.542. **Augmented data**: The gold data was substantially expanded using four data augmentation strategies, each designed to simulate the kinds of variation found in historical Tibetan manuscripts:55   - **Random noise injection**: Probabilistic character substitutions, diacritic variations, and orthographic inconsistencies calibrated to realistic manuscript variation frequencies (following [Huang et al. 2023](https://www.isca-archive.org/sigul_2023/huang23_sigul.html)).56   - **OCR-based noise simulation**: OCR-realistic noise patterns generated using the [nlpaug](https://github.com/makcedward/nlpaug) library.57   - **Rule-based diplomatic transformations**: Stochastic application of character replacements reflecting common scribal conventions in historical Tibetan manuscripts.58   - **Dictionary-based augmentation**: Insertion of entries from a [custom Tibetan abbreviation dictionary](https://huggingface.co/datasets/pagantibet/Tibetan-abbreviation-dictionary) (~10,000 abbreviation–expansion pairs) to help the model learn abbreviation resolution.59 60Additional training data was derived from the Standard Classical Tibetan ACTib corpus (>180 million words; [Meelen & Roux 2020](https://zenodo.org/records/3951503)), processed into manuscript-length lines using the `createTiblines.py` script.61 62Full details of the data preparation and augmentation pipeline are described in the [GitHub repository](https://github.com/pagantibet/normalisation).63 64---65 66## Intended Use67 68This model is intended for:69 70- **Normalisation of diplomatic Classical Tibetan texts** into Standard Classical Tibetan, as a preprocessing step for downstream NLP tasks (e.g. tokenisation, tagging, translation).71- **Digital humanities** workflows for processing historical Tibetan manuscripts, particularly texts with heavy abbreviation or non-standard orthography.72- **Research** on low-resource historical text normalisation.73 74**Note on pipeline order**: Per the results in Meelen & Griffiths (2026), it is recommended to apply normalisation *before* tokenisation in the processing pipeline. For challenging diplomatic texts, combining this model with the KenLM n-gram ranker and rule-based pre/post-processing (see [Inference](https://github.com/pagantibet/normalisation/tree/main/Inference)) yields the best results.75 76---77 78## How to Use79 80The model can be used with the inference scripts provided in the [PaganTibet normalisation repository](https://github.com/pagantibet/normalisation/tree/main/Inference). Six inference modes are available, ranging from rule-based only to combined neural + n-gram + rule-based pipelines:81 82```bash83# Run on a GPU cluster via Slurm84sbatch tibetan-inference-flexible.sh85 86# Or run directly87python3 tibetan-inference-flexible.py88```89 90See the [Inference ReadMe](https://github.com/pagantibet/normalisation/blob/main/Inference/FlexibleInference_ReadMe.md) for full usage details and configuration options.91 92---93 94## Evaluation95 96The training script includes a built-in beam search evaluation. Separate evaluation is available via the evaluation scripts, which reports:97 98- **CER** (Character Error Rate)99- **Precision, Recall, F1**100- **Correction Precision (CP) and Correction Recall (CR)** (following [Huang et al. 2023](https://www.isca-archive.org/sigul_2023/huang23_sigul.html)) for a more accurate picture of normalisation effectiveness101- **Bootstrapped Confidence Intervals** (1,000 iterations) for small test sets (optional)102 103Two versions of the evaluation script are available:104 105- evaluate_model.py — the standard script106- evaluate-model-withCIs.py — an extended version that additionally computes 95% bootstrap confidence intervals (CI) for all metrics107 108```bash109sbatch evaluate-model.sh110# or111python3 evaluate_model.py112# or113python3 evaluate-model-withCIs.py114```115 116Full evaluation results including confidence intervals and example predictions are available in the [non-tokenised Evaluations directory](https://github.com/pagantibet/normalisation/tree/main/Evaluations/Gold-nontokenised-CI) of the repository.117 118---119 120## Related Models and Resources121 122All models and datasets from the PaganTibet normalisation project are collected in the [Normalisation collection on Hugging Face](https://huggingface.co/collections/pagantibet/normalisation).123 124| Resource | Link |125|---|---|126| Training dataset | [`pagantibet/normalisation-S2S-training`](https://huggingface.co/datasets/pagantibet/normalisation-S2S-training) |127| Abbreviation dictionary | [`pagantibet/Tibetan-abbreviation-dictionary`](https://huggingface.co/datasets/pagantibet/Tibetan-abbreviation-dictionary) |128| Training & inference code | [github.com/pagantibet/normalisation](https://github.com/pagantibet/normalisation) |129| ACTib corpus | [Zenodo (Meelen & Roux 2020)](https://zenodo.org/records/3951503) |130| PaganTibet project | [pagantibet.com](https://www.pagantibet.com/) |131 132---133 134## License135 136This model is released under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). It may be used freely for non-commercial research and educational purposes, with attribution and under the same licence terms.137 138---139 140## Funding141 142This work was partially funded by the European Union (ERC, Pagan Tibet, grant no. 101097364). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency.