rafaelpo/everyrow_active_learning

How to Replace Human Annotators with LLMs for Data Labelling

Data labelling is the bottleneck in most ML projects. This interactive tutorial shows how to replace human annotators with automated LLM labelling using **everyrow** — achieving equivalent classifier performance at $0.26 per run.

What is everyrow?

everyrow is a Python SDK for running LLM operations over dataframes — screen, rank, dedupe, merge, and agent tasks. In this tutorial we use everyrow.agent_map, which applies a structured LLM call to every row in a DataFrame and returns validated Pydantic objects.

How the Data Labelling Pipeline Works

1. 1.Train a text classifier (TF-IDF + LightGBM) on a small seed dataset
2. 2.Use entropy-based uncertainty sampling to find the examples the model is most confused about
3. 3.Send those examples to everyrow.agent_map for LLM labelling with Pydantic structured outputs — no parsing, no cleanup
4. 4.Add the LLM-labelled examples to the training set and retrain
5. 5.Repeat for 10 iterations (200 labels total)

LLM Labels Match Human Accuracy — Within 0.1% Across 10 Runs

From 10 independent repeats on DBpedia-14 (14-class text classification, 200 samples labelled per run):

LLM label accuracy: 96.1% ± 1.6% agreement with ground truth labels. Roughly 1 in 25 labels disagrees with the human annotation, but that doesn't hurt the downstream classifier.

Why Use LLM Data Labelling?

• —No human annotators needed — LLM labels match human-label classifier performance
• —Structured outputs — Pydantic response models guarantee valid labels, no post-hoc parsing
• —Cost-effective — $0.26 per run ($0.0013 per labelled item)
• —Fast — label 200 samples in under 5 minutes vs hours/days for human annotation
• —Trackable — everyrow.session groups related API calls for debugging and cost tracking

Getting Started

1. 1.Get a free API key from everyrow.io/api-key ($20 free credit)
2. 2.Open active_learning_tutorial.ipynb
3. 3.Set your API key and run all cells

Also Available On

• —Kaggle Notebook — run directly in your browser, no setup needed

Resources

• —everyrow SDK on GitHub
• —everyrow documentation & API keys
• —DBpedia-14 dataset