CodeSoulco/TextInsightBench

1# Usage2 3**English** | [简体中文](USAGE.zh-CN.md)4 5## 1. Install and download6 7Follow the [quick start](https://github.com/erwinmsmith/TextInsightBench#quick-start). Run commands from the cloned repository root. Python 3.10+ is required; `pip install -e .` installs the runner and evaluator.8 9`tib download --output data/participant` downloads questions and all 50 task corpora, using the exact dataset commit in `benchmark/data.lock.json`. It does not download the optional learning pool.10 11To use the learning pool:12 13```bash14pip install -e '.[data]'15tib download --output data/participant --with-learning16tib verify --data data/participant --with-learning17```18 19The pool is Parquet; task corpora are gzip JSONL. See [fields and counts](DATA.md).20Evaluation assets are optional and public:21 22```bash23tib download --organizer --output data/evaluation24```25 26Current discovery scoring does not need a reference file. If you supply `--references data/evaluation/references.json`, use the same argument for both `judge` and `evaluate`.27 28## 2. Implement the interface29 30Your executable receives one JSON object on stdin:31 32```json33{34  "task": {"task_id": "...", "question": "...", "kind": "group_difference"},35  "corpus": {36    "path": "/absolute/path/corpus.jsonl.gz",37    "format": "jsonl.gz",38    "n_documents": 10000,39    "sha256": "..."40  },41  "learning_directory": null42}43```44 45The actual task includes all allowed metadata, population sizes and submission constraints. Read the corpus in your own tools; do not expect inline documents. Use original document IDs and text when producing assignments and exact quotation offsets.46 47Return a single JSON object on stdout matching [the submission contract](SUBMISSIONS.md). Emit diagnostics only to stderr. An empty finding list is an abstention, not a claim that no useful pattern exists. The bundled abstaining agent is only an interface example; no mining solver is bundled.48 49## 3. Test one task, then run the corpus50 51```bash52tib run --data data/participant --command 'python my_agent.py' \53  --limit 1 --timeout 3600 --output runs/pilot/submissions54 55tib run --data data/participant --command 'python my_agent.py' \56  --timeout 3600 --output runs/full/submissions57```58 59Use `--task-id TASK_ID` to select a particular task. For pool-assisted learning, add `--track unlabeled_pool`. The timeout is per task.60 61The runner launches a fresh agent process per task, checks its submission and stores valid results. Repeating the same command with the same configuration and output directory reuses valid submissions. A changed command, task selection, timeout or track requires a fresh output directory. In particular, a one-task pilot and a full run must use different directories.62 63The process runner is not a sandbox. Isolate generated code, restrict network and resource access, and keep credentials outside the execution kernel. Never expose evaluation feedback to the solver. Treat corpus text as untrusted data, not instructions.64 65## 4. Review and score66 67Export `JUDGE_API_KEY`, `JUDGE_BASE_URL` and `JUDGE_MODEL` locally. The endpoint must support JSON responses through chat completions; include `/v1` in the base URL if your provider requires it. The CLI reads environment variables, not an `.env` file automatically.68 69```bash70tib judge --data data/participant --submissions runs/full/submissions \71  --base-url "$JUDGE_BASE_URL" --model "$JUDGE_MODEL" \72  --audit-documents 160 --max-output-tokens 12000 \73  --output runs/full/reviews74 75tib evaluate --data data/participant --submissions runs/full/submissions \76  --reviews runs/full/reviews --output runs/full/report.json77```78 79Judging incurs model charges and has no automatic spending cap. Start with `--limit 1` or `--task-id TASK_ID` on the judge command to inspect cost and compatibility. A 160-document audit requires at least 14 blind-check requests before narrative grading; long documents and permitted repairs can increase this. Abstentions make no model calls.80 81Completed compatible reviews are reused on rerun. Keep `judge_config.json` and the bound `.blind.json` caches. Configuration changes require a fresh review directory. A failed judge request is not a zero score; fix service or format compatibility and rerun. Do not silently coerce semantic labels or select the highest-scoring retry.82 83`evaluate` writes JSON and Markdown reports and does not overwrite existing reports. Choose a fresh report filename when recomputing. It evaluates the full task inventory: a one-task pilot leaves the other 49 tasks missing.84 85## 5. Read the report86 87| Metric | Meaning |88|---|---|89| `scored_tasks / tasks` | Coverage with numerical task scores |90| `quality_mean` | Available only if every task has a numerical score |91| `conditional_quality_mean` | Mean over scored tasks only; never hide its denominator |92| `valid_submission_rate` | Structurally valid submissions, including abstentions |93| `abstention_rate` | Valid submissions that decline to report findings |94| `pending_tasks` | Reviews pending or evidence unresolved; inspect per-task status |95| `missing_tasks`, `invalid_tasks` | No submission or a submission that fails validation |96| `reference_coverage_mean` | Unavailable for the current tasks, which have no fixed references |97 98Scores range from 0 to 100. Zero means a scored but unsupported/unfulfilled finding, not a missing run. A task score averages its submitted findings; any unresolved finding makes task quality unavailable.99 100Keep the code commit, data lock, agent commit/configuration, prompts, model IDs, resource budgets and review configuration with your report. Freeze a configuration before making controlled comparisons.101