SimoneAstarita/Pride-large-try-sweep-20251008-183651-t00

Pride-large-try-sweep-20251008-183651-t00

Multilingual XLM-T (EN/IT/ES) binary classifier Task: LGBTQ+ reclamation vs non-reclamation on social media text.

Trial timestamp (UTC): 2025-10-08 18:36:51 Data case: es-it

Configuration (trial hyperparameters)

Dev set results (summary)

Thresholds

• —Default: 0.5
• —Best global: 0.8
• —Best by language: { "it": 0.75, "es": 0.8 }

Detailed evaluation

Classification report @ 0.5

precision recall f1-score support

hate (0) 0.9670 0.8402 0.8991 244 recl (1) 0.5301 0.8627 0.6567 51

accuracy 0.8441 295 macro avg 0.7486 0.8515 0.7779 295 weighted avg 0.8915 0.8441 0.8572 295

Classification report @ best global threshold (t=0.80)

precision recall f1-score support

hate (0) 0.9380 0.9303 0.9342 244 recl (1) 0.6792 0.7059 0.6923 51

accuracy 0.8915 295 macro avg 0.8086 0.8181 0.8132 295 weighted avg 0.8933 0.8915 0.8923 295

Classification report @ best per-language thresholds

precision recall f1-score support

hate (0) 0.9380 0.9303 0.9342 244 recl (1) 0.6792 0.7059 0.6923 51

accuracy 0.8915 295 macro avg 0.8086 0.8181 0.8132 295 weighted avg 0.8933 0.8915 0.8923 295

Data

• —Train/Dev: private multilingual splits with ~15% stratified Dev (by (lang,label)).
• —Source: merged EN/IT/ES data with bios retained (ignored if unused by model).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig
import torch, numpy as np

repo = "SimoneAstarita/Pride-large-try-sweep-20251008-183651-t00"
tok = AutoTokenizer.from_pretrained(repo)
cfg = AutoConfig.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

texts = ["example text ..."]
langs = ["en"]

mode = "best_global"  # or "0.5", "by_lang"

enc = tok(texts, truncation=True, padding=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits
probs = torch.softmax(logits, dim=-1)[:, 1].cpu().numpy()

if mode == "0.5":
    th = 0.5
    preds = (probs >= th).astype(int)
elif mode == "best_global":
    th = getattr(cfg, "best_threshold_global", 0.5)
    preds = (probs >= th).astype(int)
elif mode == "by_lang":
    th_by_lang = getattr(cfg, "thresholds_by_lang", {})
    preds = np.zeros_like(probs, dtype=int)
    for lg in np.unique(langs):
        t = th_by_lang.get(lg, getattr(cfg, "best_threshold_global", 0.5))
        preds[np.array(langs) == lg] = (probs[np.array(langs) == lg] >= t).astype(int)
print(list(zip(texts, preds, probs)))

### Files
reports.json — all metrics (macro/weighted/accuracy) for @0.5, @best_global, and @best_by_lang.
config.json — stores thresholds: default_threshold, best_threshold_global, thresholds_by_lang.
report_0.5.txt, report_best.txt — readable classification reports.
postprocessing.json — duplicate threshold info for external tools.