CodeSoulco/TextInsightBench

1# Exploration challenge2 3The default 50 tasks require broad corpus exploration, not an additional profile.4Each contains 5,000 or 10,000 documents. The agent selects its text conditions,5population, group values or time boundary. Research objectives cover practical6tradeoffs, process breakdowns, adaptation burdens, recurrence and consequences7without prescribing which pattern exists.8 9The difficulty is in finding a substantive relationship, operationalizing it10across the analysis population, and explaining its scope and competing accounts.11Corpus size alone is not evidence of a difficult or valid evaluation.12 13A selected population must contain at least 500 documents for App Reviews and141,000 otherwise. Each metadata/time comparison arm needs at least 50 or 10015documents respectively. These thresholds refer to total documents, not positive16cases. Unknown labels must be retained and interpreted. At most three17nonredundant findings are accepted. Tiny handpicked groups and ID/text-based18population filters are prohibited.19 20The fixed audit axes are entity_id, rating and report_year. Missing or degenerate21axes remain unavailable, never silently replaced with a favorable axis. In22particular, stratifying on the same metadata as the comparison may leave no23comparable strata; report that limitation. A correctly established composition24effect or reversal can earn full credit. Stability in every axis is not required.25 26## Exact audit calculations27 28All base statistics remain required. Each finding additionally reports the29`robustness_*` numeric/null fields returned by `expected`:30 31```python32from textinsightbench.validation import expected33from textinsightbench.difficulty import audit34 35# The runner's task already includes the discovery protocol. The agent supplies36# definitions, all document assignments, evidence and interpretation itself.37finding['statistics'] = expected(finding, task, documents)38statistics, stratum_details = audit(finding, task, documents)39```40 41For group/time tasks, arm 0 and arm 1 are the agent's declared comparison groups,42excluding unknown condition assignments. For compound tasks, arm 0 is known43not-A and arm 1 is known A, restricted to known B; the outcome is B. Thus the44effect always matches the sign convention of the declared comparison.45 46For each axis:47 481. A comparable stratum has at least 5 known documents in **each** arm. Its effect49   is `100 × (positive_rate_arm1 − positive_rate_arm0)`.502. The standardized effect is the mean of eligible stratum effects, weighted by51   their pooled known-arm counts. `covered_crude_delta_pp` recomputes the crude52   contrast on those same eligible rows, so differences in covered populations53   are not mistaken for reversals. Report eligible/excluded known counts and the54   minimum and maximum stratum effects.553. Select the largest nonmissing stratum by its full document count, independently56   of labels; ties use lexical order of its string value. Remove it and missing57   metadata rows. The retained effect is null unless each arm still has at least58   5 known documents. Report removed and retained-arm counts.594. `max_support_share` is the largest stratum's share of positive outcomes among60   known-arm documents with nonmissing metadata. Report positive outcomes with61   missing metadata separately. For compound tasks this is B concentration in the62   jointly known population, not the proportion of A-and-B quotes.63 64All expected fields must be present, no extra statistics are accepted, and values65must match recomputation within 1e-6. The three supporting quotes must reference66positive documents (joint positives for compound findings). A counterexample must67reference a known negative or, for compound tasks, a known discordant case. The68semantic judge still checks whether assignments and quotes are actually correct.69 70The audit does not create independent samples, establish causal identification,71eliminate multiple-testing risk, or act as unseen holdout confirmation. Unknown72assignment bounds remain mandatory for group/time findings. Compound findings73must interpret the jointly unknown population and avoid unsupported extrapolation.74 75 76## Evaluation77 78Quality is support × (15 + 25S + 20E + 30D + 10C). Depth requires a useful finding79and a supported discussion of alternatives and search/selection bias. See80[scoring](SCORING.md). Numeric audit correctness does not establish semantic81label correctness: the current judge audits a document sample, with uncertainty82preserved. No independent validation phase or causal identification is implied.83 84Current tasks automatically select this protocol. Do not use --difficulty hard.85The CLI retains historical profile support solely for historical task snapshots.86Runs, reviews and scores are bound to the current task hashes.87