TonarTechnologies/sentiment-xlmr-xl-lora

sentiment-xlmr-xl-lora

LoRA adapter reproducing Tonar's production mention sentiment model -- a full fine-tune of XLM-RoBERTa-XL -- at matched quality, so sentiment can share the multi-LoRA serving path instead of needing its own dedicated XL deployment.

Two heads over a masked-mean-pooled encoder: app_head (applicability, 1 logit) and cls_head (7 ordered sentiment classes). The score is the class distribution's expectation over anchors evenly spaced on [-1, 1].

Results

695-item blessed validation fixture, against the same gold targets the full fine-tune is scored on:

MAE is 0.0033 worse and rank correlation 0.0052 better -- indistinguishable on 695 items. Parity is against the gold targets, which is the decision-grade comparison; the blessed |delta-score| tolerances (mean <= 0.005) exist for identical weights on a different GPU and a distilled LoRA is not expected to meet them.

Caveat that matters for cutover: mean |delta-score| vs the full fine-tune is 0.056 (p95 0.214), and 28 of 695 items (4%) land on the other side of a gate threshold. Scores agree closely on average, but items near a category boundary can flip. Verify threshold calibration before switching production over.

Training

Distilled from the full fine-tune's own outputs on 116,508 inputs rendered from production mention-extraction traces: BCE against the teacher's applicability plus KL against its class distribution. LoRA r=64, 2 epochs, ~4h on an A100-80GB. Final val BCE 0.454, KL 0.054.

Two things that cost time and are worth knowing:

• —The embedding must sit in its own optimizer group with `weight_decay=0`. AdamW's decoupled weight decay is applied to every parameter in a group whether or not it has a gradient, so a resized embedding silently decays toward zero.
• —A BCE that parks at 0.695 is not necessarily a bug. H(0.532) = 0.6911 -- that is the head having learned the marginal and nothing more. Check the base rate before debugging.

Use

from peft import PeftModel
model.encoder = PeftModel.from_pretrained(model.encoder, "TonarTechnologies/sentiment-xlmr-xl-lora")
heads = torch.load("heads.pt")          # app_head + cls_head, shipped alongside

heads.pt is required -- the two heads are outside the adapter and randomly initialised otherwise. Inputs are rendered exactly as infer_standalone.py does; a different rendering is a silent accuracy loss, not an error.

Validated scope

Topic generalisation was measured for the sibling mention models and split by task: the title/description model transferred without loss, the coverage model did not (recall 0.52). Neither outcome should be assumed here without measuring.

Limitations

• —Swedish-dominant training data; multilingual by inheritance from XLM-R.
• —XLM-R caps at 514 position embeddings at every size (base, large, XL). Long inputs must be windowed, not merely truncated by the tokenizer.
• —The 4% gate-flip rate above is the one open item before production cutover.