Swift-Support/labse-intent-1.0

Swift-Support LaBSE Intent Classifier (v1.0)

Fine-tuned LaBSE for 77-way banking intent classification across English, Sinhala, Tamil and their romanized code-mixed forms (Singlish, Tanglish). Part of the Swift support-ticket triage project.

Card revised 2026-09-08. The previous version reported a 3-epoch training run and explained the Indic specialists' failure with a mechanism the data does not support. Both are corrected below, and the correction is kept visible rather than quietly overwritten. See What changed and why.

Model details

The split matters and is not the official BANKING77 split. The same ticket exists five times under one id. Scores here are macro-F1 pooled over all five tracks of the frozen split, so they are not comparable to BANKING77 leaderboard numbers, which are English-only accuracy on the official split.

Evaluation

Six encoders, identical budget (6 epochs, lr 2e-5, batch 32, max length 128, fp16, seed 42), fitted on train and scored on dev.

Pooled and per-track macro-F1 (dev, %)

LaBSE and mmBERT are statistically indistinguishable on this task. Paired bootstrap over 1,000 resamples: Δ = −0.0001, 95% CI [−0.0045, +0.0046], p = 0.944. This card previously called LaBSE "the intent champion". It is not; it ties. Either model is a defensible choice, and mmBERT is the smaller of the two.

These are dev numbers. The held-out test numbers are now available — see below.

Held-out test results

Fitted on train+dev (49,990 rows), scored once on the frozen split's test set (15,395 rows). epoch_selection = final-epoch with best_epoch = 6 of 6 for all three, so no epoch-selection bias is possible.

On dev, mmBERT and LaBSE tied and XLM-R trailed. On test the order is LaBSE > XLM-R > mmBERT. This is a point estimate, not a tested difference — that run wrote no per-row predictions, so no paired test is possible. Do not read it as mmBERT being beaten.

⚠️ The tamilish column is a corpus defect, not a model result

Tamilish scores 69.28 on test against 91.70 on dev, a drop no other track shows (english, sinhala and tamil move 1–3 points; singlish 2). The cause is a vocabulary discontinuity between the train and test renderings of that one track:

The tamilish rendering of the BANKING77 test file was produced by a different transliteration process from the tamilish rendering of the train file. Tamilish test text also averages more characters but fewer words than tamilish train, and is the only track with duplicate test strings.

Do not quote 69.28 as this model's ability to handle romanized Tamil. It measures a train/test mismatch in the corpus. The effect is task-dependent — priority on tamilish is 0.8142 against english's 0.9229, a gap rather than a collapse — because a 3-way task tolerates vocabulary drift that a 77-way task does not. This is being fixed for corpus v1.1.

What changed and why

1. The old table was trained for 3 epochs and understated the whole roster. Five of six models were still improving when training stopped at 6, so even these numbers are a lower bound. The effect on MuRIL was severe — its Tamil score was reported at 66.01% and is 90.09% at an adequate budget.

2. The old explanation of the Indic specialists' failure was backwards. The previous card said MuRIL and IndicBERT "failed outright on the pooled and code-mixed tracks because their smaller vocabularies couldn't handle heavy romanization or English slang".

The data says the opposite. MuRIL's romanized tracks are among its best: Tanglish 90.23, Singlish 88.79. It fails on exactly one track — native-script Sinhala, at 76.33, 13.7 points below its own English. The mechanism is in its tokenizer:

MuRIL maps 64.5% of Sinhala characters to `[UNK]` — the characters are destroyed before the first transformer layer, and no amount of fine-tuning recovers them. Every other model in the roster is at 0.0% on every track.

So the correct claim is not "specialists cannot handle romanization". It is "check the tokenizer's `[UNK]` rate on your script before fine-tuning anything". Romanization is handled fine by every model here; script coverage is what varies.

Note also that token fertility (tokens per word) does not predict downstream quality across this roster — Spearman ρ = −0.035 against per-language deficit, versus −0.521 for [UNK] rate. Fertility is a cost; [UNK] is destruction. They are often conflated.

Intended use and limitations

Intended: routing inbound banking support tickets to one of 77 intents, as a triage aid with a human in the loop.

Out of scope: non-banking domains (the taxonomy and vocabulary are banking-specific); languages or scripts outside the five tracks; any customer-facing decision taken without review.

Performance is not uniform across tracks. Tanglish trails English on every system measured. Expect worse service for customers writing in transliteration — which, in this setting, correlates with who those customers are.

Intent labels are BANKING77's human gold, unlike this project's sentiment and priority labels, which are LLM-generated. That makes intent the most trustworthy of the three tasks here.

Usage

from transformers import pipeline

classifier = pipeline("text-classification", model="Swift-Support/labse-intent-1.0")
print(classifier("I lost my credit card yesterday, please help me cancel it"))

Citation

Citing BANKING77 alone is not sufficient attribution for this corpus. BANKING77 supplies the English text and the 77-way intent labels — 20% of the rows and one of the three label columns. The rest is this project's work:

If you use the translated tracks, the sentiment or priority labels, the frozen split, or any of the models, cite this work. If you additionally use the English text or the intent labels, cite BANKING77 as well. Both licences are CC-BY-4.0, so attribution is a licence condition, not a courtesy.

This work — the corpus

@misc{swift_tickets_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift Support Tickets: A Multilingual Ticket-Triage Corpus for Sinhala,
                  Tamil and English, Including Romanized Code-Mixed Text},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Swift-Support/swift-support-tickets-1.0}},
  note         = {Derived from BANKING77. Adds four language tracks and the sentiment and
                  priority label columns. Paper in preparation.}
}

This work — the models

@misc{swift_models_2026,
  author       = {Sithija Seneviratne and Ruththiragayan Sutharsan and Shazan Shaheed},
  title        = {Swift: Multilingual Baselines for Support-Ticket Triage in Sinhala,
                  Tamil and English},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Swift-Support}},
  note         = {Intent, sentiment and priority classifiers. Paper in preparation.}
}

BANKING77 — the source of the English text and the intent labels

@inproceedings{casanueva2020banking77,
  author    = {Casanueva, I{\~n}igo and Tem{\v c}inas, Tadas and Gerz, Daniela and
               Henderson, Matthew and Vuli{\'c}, Ivan},
  title     = {Efficient Intent Detection with Dual Sentence Encoders},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
               Conversational AI},
  pages     = {38--45},
  year      = {2020},
  publisher = {Association for Computational Linguistics},
  doi       = {10.18653/v1/2020.nlp4convai-1.5},
  url       = {https://aclanthology.org/2020.nlp4convai-1.5/}
}