sweenk/snt-classifier

SNT News Classifier v0.6

Multi-label news topic classifier: 12 top-level (L1) and 71 sub-level (L2) categories, built on xlm-roberta-large with two independent sigmoid heads. Both levels are genuinely multi-label — an article about a trade deal can be world + money_and_business + politics at the same time. Per-class decision thresholds (tuned on a held-out validation split) ship inside config.json; predict_labels() applies them and falls back to argmax so no article is ever left unlabeled.

Built by Sweenk to categorize its news feed; released so others can use and scrutinize it.

Links: GitHub — model & training code · GitHub — data pipeline

At a glance

• —Multi-label at both levels — a story can be world + money_and_business + politics at once, each above its own per-class tuned threshold.
• —0.847 L1 macro-F1 on 26,412 held-out articles; all 12 top-level categories clear a 0.65 per-class floor.
• —Multilingual encoder — xlm-roberta-large (100 languages), fine-tuned on ~264K news articles.
• —Honest about its labels — the gold is LLM-teacher-generated, not human-annotated, and the evaluation section says exactly what that means.

Quick start

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("sweenk/snt-classifier", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("sweenk/snt-classifier")

enc = tok("OpenAI raises $6.6B. The startup announced its latest funding round...",
          return_tensors="pt", truncation=True, max_length=512)
print(model.predict_labels(**enc))
# [{'l1': [{'key': 'money_and_business', 'p': 0.99}, {'key': 'tech_and_ai', 'p': 0.98}],
#    'primary_l1': 'money_and_business',
#    'l2': [{'key': 'companies_and_industries', 'p': 0.92}, ...]}]

Input convention: "{title}\n\n{body}", truncated at 512 tokens. The classifier was trained on title+body; titles alone work but body text improves routing (the training prompt explicitly prioritizes body over headline).

Taxonomy — 12 L1 / 71 L2

The full machine-readable taxonomy (l1_keys, l2_keys, l2_parent, per-class thresholds) is in config.json.

Evaluation — our own numbers, stated plainly

Held-out test split: 26,412 articles (a 10% slice of the labeled corpus, stratified by primary L1). "Tuned" = per-class thresholds optimized on the validation split, then applied unchanged to test.

Per-class L1 F1 (test)

What you should know before trusting these numbers

Read this section — it is the honest part.

• —The gold labels are model-assisted, not human-annotated. The corpus (~278K articles from HuffPost archives, CommonCrawl News, daily.dev, and Sweenk production) was labeled by a mechanical migration from an earlier taxonomy plus multiple passes of a Claude Sonnet teacher with a rule-based prompt, spot-audited by humans (QA gates at 70–87% agreement on sampled batches). Test F1 therefore measures agreement with an LLM teacher, not with human ground truth.
• —Class imbalance is real (~21x). politics/lifestyle/entertainment have ~44-47K training rows; science_and_space ~2.2K and tech_and_ai ~3.7K. Training compensates with per-class pos_weight (clamped at 10) and caps the majority classes at 25K primary-label train rows so they don't swamp the rare ones. The weakest class here is human_stories (F1 0.671) — the fuzziest category by construction.
• —Labels were corrected over time, so F1 is not comparable across releases. v0.5.1 re-labeled ~1,500 systematically mislabeled rows with route-by-cause rules (accidents by cause, terror → crime, pharma earnings → money); on those rows agreement with the corrected gold went 25.9% → 75.3%. Because the gold labels themselves changed between releases, aggregate F1 deltas are not like-for-like — treat each release's numbers as self-referential.
• —Multilingual ability is inherited, not measured. The encoder is XLM-R, but nearly all training articles are English. Expect degraded (unquantified) quality on non-English news.
• —L3 (named topics / entities) is not part of this model — Sweenk handles that downstream with a separate extraction step.

Architecture

xlm-roberta-large encoder → CLS pooling → dropout(0.1) → two parallel linear heads (L1: 12 logits, L2: 71 logits), both sigmoid. Trained 3 epochs, BCE loss with per-class pos_weight (L1) and loss weights L1:1.0 / L2:2.0, bf16 autocast, gradient checkpointing. Inference upcasts logits to fp32 before sigmoid (bf16 sigmoid saturates above logit ~6.2, which collapses co-confident multi-label pairs).

Versions

License & attribution

Model weights: MIT. Base model: FacebookAI/xlm-roberta-large (MIT). The training corpus contains article text from public news sources and is not redistributed with this model.