Datasetpublic

anote-ai/annotatebench-results

AnnotateBench Results AnnotateBench is a cost-aware benchmark for comparing annotation strategies across public text-classification datasets and label budgets. This repository contains derived experiment results and aggregate statistical summaries. It does not redistribute the source datasets. Before making this repository public: replace this notice after the Anote AI team has selected a release license and verified that the derived results may be distributed under that… See the full description on the dataset page: https://huggingface.co/datasets/anote-ai/annotatebench-results.

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes45downloads

Dataset Card

AnnotateBench Results

AnnotateBench is a cost-aware benchmark for comparing annotation strategies across public text-classification datasets and label budgets. This repository contains derived experiment results and aggregate statistical summaries. It does not redistribute the source datasets.

Before making this repository public: replace this notice after the Anote AI team has selected a release license and verified that the derived results may be distributed under that license. Do not add a license field to the metadata until that decision is complete.

Paper and code

—Paper: arXiv URL pending; the submitted manuscript source is available in the code repository.
—Code: https://github.com/anote-ai/Research-AnnotateBench
—Organization: Anote AI

For exact commands and claim boundaries, see README.md, ARTIFACT.md, and results/README.md in the code repository.

Benchmark scope

The primary benchmark covers ten public text-classification datasets: Financial PhraseBank, TREC, Banking77, AG News, SST-2, 20 Newsgroups, Rotten Tomatoes, Yelp Polarity, TweetEval Sentiment, and Emotion.

The gold-label experiment compares random, uncertainty, diversity, and hybrid selection at budgets of 50, 100, 250, 500, and 1,000 labels. The primary downstream model is TF-IDF plus logistic regression. The repository also contains a sentence-transformer robustness comparison and an API-backed LLM annotator extension.

The expanded release adds a five-seed sentence-embedding grid, paired top-two strategy statistics, nested cumulative trajectories on three representative datasets, and an AG News candidate-pool scalability study. The scalability measurements are implementation- and machine-specific; they are not claims of distributed-system throughput.

The gold-label benchmark simulates annotation by revealing labels already present in public datasets. It is not a release of newly collected human annotations.

Files

Configuration	File	Description
`benchmark_results`	`data/benchmark_results.csv`	Primary gold-label benchmark grid, including performance and human-cost scenarios.
`benchmark_results_cost_unified`	`data/benchmark_results_cost_unified.csv`	Unified gold-label and LLM-annotator cost results.
`paper_core_summary`	`data/paper_core_summary.csv`	Best strategy-budget summary used by the paper.
`budget_recommendations`	`data/budget_recommendations.csv`	Lowest-cost strategy and budget meeting each reported target.
`downstream_model_comparison`	`data/downstream_model_comparison_summary.csv`	TF-IDF and sentence-transformer robustness comparison.
`sentence_transformer_five_seed_grid`	`data/benchmark_results_sentence_transformer_logreg_5seed.csv`	Full five-seed sentence-transformer robustness grid with three cost scenarios.
`downstream_model_condition_summary`	`data/downstream_model_condition_summary_5seed.csv`	Five-seed aggregates by dataset, representation, strategy, and budget.
`downstream_model_robustness`	`data/downstream_model_robustness_summary.csv`	Dataset-level representation sensitivity and best-strategy agreement.
`paired_strategy_statistics`	`data/paired_strategy_statistics.csv`	Paired top-two differences, bootstrap intervals, effect sizes, sign-flip tests, and Holm adjustment.
`cumulative_trajectories`	`data/cumulative_trajectory_results.csv`	Nested acquisition results for Financial PhraseBank, TREC, and Yelp Polarity.
`pool_scalability`	`data/pool_scalability_results.csv`	AG News selection runtime and utility at candidate pools of 1,200, 5,000, and 10,000.
`statistical_appendix`	`data/statistical_appendix.csv`	Aggregate row-level reliability diagnostics and confidence intervals.
`statistical_significance`	`data/statistical_significance.csv`	Statistical comparisons with multiple-testing correction.

The exact columns differ by configuration. Column definitions and generation logic are documented in results/README.md and the scripts in the code repository.

Cost interpretation

Human annotation costs are scenario estimates derived from public pricing assumptions, not observed invoices. LLM API costs use recorded token counts for seeds 1 and 2 and a validated stratified estimate for seed 0. They exclude human review, failed requests, latency, retries, and quality-control work.

Limitations

—The benchmark is classification-only.
—Public gold labels simulate annotation and do not capture annotator disagreement, fatigue, interface effects, or adjudication.
—The primary downstream classifier is intentionally controlled and is not a state-of-the-art performance claim.
—Several small strategy differences have overlapping confidence intervals.
—None of the ten five-seed exact top-two comparisons remains significant after Holm correction; the paired analysis is exploratory.
—The LLM extension evaluates one model and prompt family and is not a complete cross-model reliability benchmark.
—The cumulative experiment covers three datasets, and the scalability study covers one dataset on one machine.

Source datasets and licenses

No source-dataset text is included here. Users must obtain each source dataset from its official distributor or existing Hugging Face dataset page and comply with its individual license and terms. Before public release, the team should complete and retain a dataset-by-dataset license review.

Citation

Add the final BibTeX entry after the arXiv submission receives an identifier.