pruthvikadam/active-learning-lab

🏷️ Active Learning Lab — hit target accuracy with far fewer labels

A data-centric ML app with two levers on the real bottleneck — labels: weak supervision (labeling functions → a label model) bootstraps a training set with zero manual annotation, and active learning (uncertainty sampling) spends human labels where they matter. The payoff is a learning curve proving it reaches target accuracy with a fraction of the labels random sampling needs.

▶️ Live demo: https://pruthvikadam-active-learning-lab.hf.space · Code: https://github.com/PruthviVKadam/active-learning-lab

Problem → Approach → Result

• —Problem: the bottleneck in most ML projects isn't the model — it's labels. Hand-labeling is slow, and labeling random examples wastes budget on cases the model already gets right.
• —Approach: two data-centric levers on the SMS Spam Collection (TF-IDF + logistic regression). (1) Weak supervision: 8 noisy labeling functions (spam keywords, URLs, shortcodes, money patterns, phone numbers, shouting; ham chat-words) each vote spam/ham/abstain; a label-free label model weights them by agreement and votes. (2) Active learning: train, query the examples the model is least sure about (least-confidence / margin / entropy), reveal those labels, retrain — vs. a random-sampling baseline. Headline metric is spam F1 (not accuracy: SMS is 87% ham, so an all-ham classifier already scores 0.87).
• —Result — averaged over 5 seeds, scored on a fixed 1,293-message held-out test (copied verbatim from `eval/results.md`):

💡 Insights gained

• —Active learning cut the labeling budget by ~5× on this task. Uncertainty sampling reached spam-F1 0.85 with 120 labels; random needed 580. The mechanism is concrete: with 12.6% spam, random labeling spends ~87% of its budget on easy ham, while uncertainty sampling keeps pulling near-boundary (often spam) messages — exactly the ones that move a decision boundary.
• —~200 labels (≈5% of the pool) matched the full-supervision ceiling. Training on all 3,876 labels gives F1 0.918; active sampling reaches 0.916–0.924 by 200–300 labels. The last 95% of labels were nearly redundant — a direct argument for labeling less, but smarter.
• —For binary classification, least-confidence, margin, and entropy are the same strategy. Their curves overlap exactly (and a unit test pins this) because all three are monotonic in |p−0.5|. They only diverge with 3+ classes — knowing that prevents a fake "we tried 3 strategies" claim.
• —A label-free label model down-weighted a bad labeling function on its own. lf_shouting (all-caps heuristic) is just 11% accurate, yet the model — which never sees a gold label — gave it weight 0.23 vs. ~0.95–1.0 for the reliable LFs, because it estimates reliability from agreement with the consensus. Weak supervision degrades gracefully when a heuristic is wrong.
• —Accuracy is the wrong headline on imbalanced data. An all-ham classifier scores 0.87 accuracy while catching zero spam. Reporting spam F1 (and starting the curve at F1 ≈ 0.05, where the seed model predicts almost all ham) is what makes the labeling progress legible.

Dataset

SMS Spam Collection (UCI; 5,574 messages → 5,169 after dedup, 12.6% spam; public, no auth). Bundled as data/sms.csv (~0.4 MB) so the demo runs instantly. Raw archive is git-ignored.

Stack

Python 3.14 · scikit-learn (TF-IDF + logistic) · numpy/scipy · Plotly · Streamlit. The labeling functions, the label model, and the active-learning loop are hand-rolled — no Snorkel, no modAL, no torch.

Reproduce

python -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt
python data.py             # load SMS Spam; fixed held-out test set
python weak_supervision.py # (imported) labeling functions + label model
python experiment.py       # active vs random learning curves -> eval/results.md
pytest -q                  # 10 tests: LF votes, label-model down-weighting, AL beats random
streamlit run app.py       # learning curves + label-budget explorer + live LF tester

Honesty guardrail

The "79% fewer labels" claim comes only from experiment.py's seeded curves over a fixed held-out test set; labeling functions are written by a human and the held-out labels are ground truth, never model-generated. Where active learning doesn't help (the first ~40 labels, where it ties random) the curve shows it.