CodeSoulco/TextInsightBench

1---2pretty_name: TextInsightBench3language:4- en5license: other6license_name: upstream-source-terms7license_link: https://huggingface.co/datasets/CodeSoulco/TextInsightBench/blob/main/SOURCES.md8tags:9- agent-evaluation10- data-mining11- evidence-grounding12size_categories:13- 1M<n<10M14task_categories:15- text-generation16configs:17- config_name: amazon_beauty_learning18  data_files:19  - split: train20    path: learning/amazon_beauty/*.parquet21- config_name: app_reviews_learning22  data_files:23  - split: train24    path: learning/app_reviews/*.parquet25- config_name: cfpb_learning26  data_files:27  - split: train28    path: learning/cfpb/*.parquet29- config_name: nhtsa_learning30  data_files:31  - split: train32    path: learning/nhtsa/*.parquet33---34 35# TextInsightBench36 37**English** | [简体中文](README.zh-CN.md)38 39A natural-language data-mining benchmark for agents: **50 tasks**, **435,000 task documents** and **944,468 unlabeled learning documents**.40 41Each task provides 5,000 or 10,000 texts and a research objective. Agents choose the patterns, populations and comparisons to investigate, then submit up to three findings with complete document assignments, exact quotations, statistics, counterexamples and limitations. Any analysis method is allowed.42 43## Contents44 45| Path | Purpose |46|---|---|47| `tasks.json` | Questions and task-specific constraints |48| `corpora/*.jsonl.gz` | Full text for each task |49| `learning/*/*.parquet` | Optional unlabeled learning pool |50| `output.schema.json` | Submission structure |51| `protocol.json`, `release.json` | Evaluation protocol and data counts |52| `manifest.json` | File hashes and sizes |53 54Sources: Amazon Beauty, Android App Reviews, CFPB and NHTSA. The task inventory contains 20 group differences, 15 temporal changes and 15 compound associations. See [data composition and fields](docs/DATA.md).55 56## Use the benchmark57 58```bash59git clone https://github.com/erwinmsmith/TextInsightBench.git60cd TextInsightBench61python -m venv .venv62source .venv/bin/activate63pip install -e .64tib download --output data/participant65tib verify --data data/participant66```67 68The code pins dataset commits in `benchmark/data.lock.json`. Add `--with-learning` when downloading for pool-assisted learning. Learning configurations also work with `datasets.load_dataset`; task corpora are accessed through the runner or as gzip JSONL.69 70[Connect an agent and score results](docs/USAGE.md) · [中文使用指南](docs/USAGE.zh-CN.md) · [Code](https://github.com/erwinmsmith/TextInsightBench) · [Evaluation assets](https://huggingface.co/datasets/CodeSoulco/TextInsightBench-Evaluation)71 72## Evaluation73 74Full assignment partitions, quotations and arithmetic are checked locally. Sampled claim-blind document checks cap subsequent finding-quality grades. Semantic review incurs model charges and is not exhaustive or independent ground truth. The tasks have no fixed reference conclusions; quality is judged against corpus evidence and the public rubric. [Scoring](docs/SCORING.md).75 76Task and learning document IDs are disjoint, but the data was previously public, entities and sources can overlap, and tasks are not statistically independent. Narratives are unverified author reports and may contain personal information. Upstream terms differ; the compilation grants no new rights over third-party text. See [source terms](SOURCES.md).77