hermanda/robeczech-propaganda

robeczech-propaganda

Fine-tuned ufal/robeczech-base classifier for pro-Kremlin propaganda in Czech/Slovak news-comment text. Comment-only input (no article context).

v2 (current): retrained on human-reviewed Argilla labels (previously: Haiku-teacher-distilled labels). Full finetune, 4 epochs, lr=2e-5, 2x positive-class oversampling.

Evaluation (605-row held-out test set, 69 positives)

v1 and v2 have near-identical AUC (ranking quality) — v1's low F1 was primarily a threshold-calibration problem, not a weaker model. v2 improves F1 mainly through better label quality and, downstream, an explicitly recalibrated operating threshold (see below).

Score calibration / recommended thresholds

Softmax score is not comparable to v1's — v2 outputs are calibrated very differently (fewer, more confident high scores). Precision/recall at threshold, measured on the same held-out test set:

Caveat: these thresholds are set from only 605 test rows (69 positives); the 0.9659 cutoff rests on just 7 true positives at that exact operating point, so treat it as directional, not exact. At this calibration, roughly 75% of actual propaganda comments fall into the "clean" tier — this is a high-precision, low-recall configuration by design (optimizing for few false alarms), not a broad detector.

Training and evaluation data

Real human-reviewed labels from an internal Argilla annotation queue (~11.4k training rows after excluding the fixed test set, 2x oversampled to ~12.7k). Comment-only input (article/parent-comment context was tested and consistently hurt performance at this scale).

Framework versions

• —Transformers 5.x, PyTorch 2.x