CodeSoulco/TextInsightBench

1# Scoring2 3The current scoring protocol is finding-quality-discovery-evidence. Commit-pinned current4scores must not be compared directly with historical narrow-task scores.5 6## Local gates7 8Each finding needs a permitted population, valid agent-selected comparison,9minimum population/arm sizes, complete condition partitions, exact text offsets,10and all recomputed statistics. Invalid submissions do not receive invented11quality scores. See [submission contract](SUBMISSIONS.md).12 13## Evidence-based quality14 15A semantic assessment supplies support, task_fulfilled, duplicate_of, rationale,16and four dimensions from 0 to 1: S statistical validity, E evidence entailment,17D analytical depth and C calibration. Dimension anchors are 1 fully justified,180.75 minor gaps, 0.5 material limitations, 0.25 weak and 0 absent/wrong.19 20```text21finding quality = support × (15 + 25S + 20E + 30D + 10C)22```23 24Support factors: supported=1, partial=0.5, unsupported=0. Uncertain stays null.25An unfulfilled task or duplicate gets 0 regardless of dimensions. Generic26sentiment, metadata frequencies or an unexamined aggregate contrast do not27fulfill the task. Depth requires a substantive discovery, a defensible choice28of scope/comparison, competing explanations, and correct interpretation of29robustness checks. Material audit omissions cap D at 0.5; merely restating30numbers caps it at 0.25. These semantic caps are judge instructions, not31deterministically proven properties.32 33Example: a supported, nonduplicate finding with S=.8, E=.9, D=.75, C=.8 earns3415+20+18+22.5+8 = 83.5. Partial support halves it to 41.75. This is an illustrative35calculation, not an observed agent result.36 37## Semantic audit38 39All selected-population arithmetic and partitions are checked. The model then40sees at most 160 documents by default: submitted quotations, samples from each41nonempty positive/negative/unknown assignment cell and a corpus-wide remainder.42The seed is created after submission and saved with judge configuration for43reproducibility; sampled IDs are retained in each review. This is a bounded44audit of submitted labels, not a new independently labeled or held-out dataset.45 46Sampled label mistakes undermine semantic support even if the counts add up.47Absence of sampled errors does not prove all labels correct. The judge must use48uncertain when the packet cannot resolve a claim. No full-corpus semantic49guarantee or unbiased estimator of label accuracy is claimed. Report judge50model, input budget and audit size; model-based scores have evaluator error.51 52Before narrative grading, a separate claim-blind pass reannotates the sampled53documents in batches of at most 12 documents / 24,000 characters (a single long54document is retained whole). It receives only condition definitions and original55text, never participant claims, labels, statistics or task questions. Exact56positive quotes and complete output coverage are checked. An invalid response57permits one format/quote repair; unresolved protocol errors stop the review.58 59The persisted blind check is bound to its prompt/protocol and the exact sampled60documents. At evaluation time, agreement diagnostics are recomputed from the61original submission and corpus. The following fixed gates cap semantic support:62 63- Unsupported: at least 5 sampled claimed positives and at least half are judged64  negative, or every supplied supporting document is contradicted by the checker.65- Uncertain: over 20% checker-unknown within any condition's sampled population,66  no checked population, or fewer than 3 supporting documents confirmed positive67  for all required conditions (unless the unsupported gate already applies).68- Partial: more than 10% disagreement among at least 20 jointly known sampled69  assignments, or over 15% positive contradictions among at least 8 sampled70  claimed positives.71- Otherwise, the check allows supported quality but does not itself award it.72 73The more restrictive support category is used, with unsupported taking priority74over uncertain and partial. Narrative grading cannot override these caps. Reports75retain the narrative support, effective_support and evidence_gate diagnostics.76These are conservative operational thresholds, not validated population-error77estimates. Sample construction is not uniform, checker judgments remain fallible,78and agreement with a model does not create independently certified ground truth.79 80A live synthetic regression check preserved full credit for a supported composition81reversal and rejected a submission labeling explicit negative texts as positive.82This is an evaluator sanity check, not a benchmark agent-performance score.83 84## Aggregation and reference availability85 86Task quality averages all submitted finding scores; any unresolved finding makes87task quality unavailable. An abstention is valid but unscored, not a verified88absence of useful findings. A full quality_mean is available only when every89task has a score. conditional_quality_mean covers only scored tasks and must be90reported alongside scored_tasks, abstention_rate, valid_submission_rate, missing,91invalid and pending counts. Source and family breakdowns are included.92 93Current tasks have no fixed reference conclusions. reference_coverage is null.94Novel supported findings are not penalized for lacking a fixed95match. Reviews bind task, corpus, submission, reference configuration and scoring96hashes; stale reviews must not be reused.97