CodeSoulco/TextInsightBench

1# Submission contract2 3Write one UTF-8 JSON file per task, named `<task_id>.json`. The agent process protocol writes that same object to stdout. Do not wrap it in Markdown.4 5```json6{7  "task_id": "the task ID",8  "findings": [],9  "abstention_reason": "No sufficiently supported finding was identified."10}11```12 13A nonempty answer contains up to three findings. Every finding has:14 15| Field | Required content |16|---|---|17| `finding_id` | Unique identifier within the answer |18| `claim` | Specific downstream conclusion, including direction and scope |19| `kind` | Exactly the task's kind |20| `scope` | Corpus, group, time and entity scope |21| `population` | Object with `filters`: zero to three conjunctive metadata rules |22| `comparison` | Agent-selected groups/date boundary, or null for association |23| `definitions` | One observable condition for group/time tasks; two for association tasks. Each has `condition_id`, `inclusion`, `exclusion` |24| `assignments` | One complete document partition per condition: `condition_id`, `positive_doc_ids`, `negative_doc_ids`, `unknown_doc_ids` |25| `statistics` | Exactly the recomputed fields below |26| `evidence` | Original quotation records: `doc_id`, `start`, `end`, `quote`, `role` (`supporting`, `counterexample`, `context`) |27| `limitations` | Nonempty list describing uncertainty, confounding and inference limits |28 29At least three distinct supporting documents and a known negative/discordant counterexample (if assigned cases exist) are required, with at most 15 spans per finding. Every selected-population document belongs to exactly one state per condition. Positive means the stated report is present; negative means it is not reported under the definition; unknown preserves unresolved judgments. Quotations refer to the `text` field. Python `text[start:end]` must equal `quote`, using Unicode characters, not UTF-8 bytes or JavaScript UTF-16 code units. Do not normalize or edit evidence text before computing offsets.30 31Population example: `{"filters":[{"field":"rating","op":"gte","value":2}]}`.32Use `{"filters":[]}` for all documents. Allowed fields are listed in each task;33operators are eq, in, gte and lte. An in list has 1–20 scalar values. Missing34metadata never matches a filter. Document IDs, text and post-hoc condition labels35cannot filter the population. Explain scope choices in scope and limitations.36 37Group example: `{"field":"rating","groups":[[1,2],[4,5]]}`. Each group has381–20 distinct values, and the groups must be disjoint. This example is a syntax39illustration, not a sufficient research finding. Group values omitted from both40arms remain in the selected population and require assignments; statistics41report their exclusion. Temporal example:42`{"field":"timestamp","cutoff":"2023-06-01"}`. Compound association uses null.43Minimum population and arm sizes are specified in each task.44 45Use the implementation to compute statistics from your assignments:46 47```python48from textinsightbench.validation import expected49finding["statistics"] = expected(finding, task, documents)50```51 52For group/time tasks, required statistics are `group0_total_n`, `group0_known_n`, `group0_positive_n`, `group0_unknown_n`, `group0_rate_known` and the corresponding five `group1_*` fields, plus `excluded_metadata_n`, `delta_known_pp`, `delta_identification_lower_pp`, `delta_identification_upper_pp`.53 54Group order follows `comparison.groups`; for time tasks, group 0 is before the cutoff and group 1 is on or after. The known rate is positive / (total − unknown). The reported difference is **group 1 minus group 0**, in percentage points. The lower/upper identification bounds allocate unknowns to all compatible states within the finite corpus. These bounds are not confidence intervals. Missing comparison metadata is excluded from named denominators but still needs a condition judgment.55 56For association tasks, required statistics are `known_joint_n`, `unknown_joint_n`, `n11`, `n10`, `n01`, `n00`, `p_b_given_a`, `p_b_given_not_a`, `conditional_difference_pp`, `lift`. Condition A is the first definition and B the second. Joint cells count documents with known judgments for both conditions; unknowns are counted separately. Zero denominators produce JSON `null`, never NaN or infinity. Additional unsupported statistics are not accepted by this contract.57 58Current findings also require corpus_total_n, population_total_n,59population_coverage and all robustness_* fields returned by expected. Pass the60entire task corpus to expected; it applies the declared population itself. Do not61pre-filter twice. See [audit formulas](DIFFICULTY.md). The schema is bundled in62textinsightbench/output.schema.json and the dataset root. Synthetic discovery63fixtures appear in tests/test_discovery.py; they are not released-task answers.64