bowang0911/statute-disease-hard-negatives

Attribute-differentiated hard negatives: statutes and disease

Three retrieval training sets built on the same principle: the negatives are near-duplicates of the gold that differ on exactly one attribute, so the only way to rank the gold first is to resolve that attribute. Topical similarity is not enough to separate them, which is the point -- a retriever trained on topically-separable data never learns fine-grained instance discrimination.

Each config is a single train split.

from datasets import load_dataset
ds = load_dataset("bowang0911/statute-disease-hard-negatives", "de_statutes", split="train")

Schema

{
  "query":          "a fact situation the gold provision governs",
  "query_short":    "the same situation in one sentence (statute configs only)",
  "query_keyword":  "keyword form of the question (disease config only)",
  "document":       "the gold passage",
  "hard_negatives": ["near-duplicate 1", "near-duplicate 2", "..."],
  "meta":           { "...": "provenance, axis labels, BM25 diagnostics" }
}

document is a string: every row here has exactly one positive. Trainers that sample one positive per row per epoch expect a list, so widen it at load time -- but the list belongs to datasets that are actually multi-positive, such as FinQA hard negatives at 1.71 golds/query, not to these.

How the negatives were built

Statutes (`de_statutes`, `uk_statutes`). For each provision, sibling provisions are drawn along five structural axes: adjacent (the section before or after), same_group (same Part/Chapter), adjacent_group, letter_sibling (same number, different letter or a same-numbered Schedule paragraph), and cross_law_heading (a different statute with an identical section heading). These are the provisions a lawyer would actually have to rule out.

Queries are then generated against those negatives, not independently. The generator sees the target and every sibling, and must write a fact situation the target governs and no sibling does. That turns "these are probably wrong" into "the narrative was written so that these are wrong". It is also forbidden from citing provision numbers, from reusing the statute's operative vocabulary, and from stating that a sibling's element is absent -- enumerating absences ("he was unarmed", "no lock was forced") is a tell that gives the answer away.

Two verification passes follow:

1. 1.Semantic, by a model from a different family than the generator, which sees the candidates shuffled and unlabelled and says which govern. The row survives only if it names exactly the target. A model is a poor judge of its own output, so same-family self-checking is the failure mode this avoids.
2. 2.Lexical, by BM25, with no API call. If the gold wins on term overlap the query leaked the statute's vocabulary and is solvable by keyword matching -- which trains nothing, because retrievers already do lexical matching well.

Disease. Passages are sections of disease articles keyed to MONDO, split by aspect (definition, symptoms, treatment, diagnosis, causes, mechanism, epidemiology, prognosis, prevention). Negatives come from two axes: other aspects of the same disease -- which is what makes it hard, since the disease name is shared and only the aspect separates them -- and the same aspect of a sibling disease.

The same-disease axis is also the dangerous one: a definition paragraph routinely restates the symptoms, so a symptoms query would train against a passage that answers it. Measured with a strong judge over 60 rows, 63.3% of rows carried at least one co-relevant negative, and all of them came from that axis. Every row was therefore passed through a judge that sees the candidates shuffled and drops the negatives which also answer the query. This removes the offending negatives and keeps the row: at a 63% row-level rate, dropping whole rows would have cost most of the dataset.

Removing the axis instead of verifying it was tried and rejected. Without it, BM25 ranks the gold first for 88.3% of rows (vs 46.5% after verification) -- different disease names in query and negative means keyword matching wins outright, and the dataset trains almost nothing.

Known limitations

Read these before training on it.

1. Negatives that are another row's gold. These sets index the same corpus repeatedly, so one row's negative is often another row's positive:

Trainers that pool hard negatives batch-wide turn this into a false negative: row A's negative becomes a negative for row B, whose gold it is. It cannot be masked by a neg > pos + threshold rule -- an identical string scores exactly equal to the positive, so equal > equal + margin is false and it trains as a negative at full weight. Add an exact-string dedup at collate time (drop any pooled negative matching a positive in the batch). Dropping them from the data instead is viable for uk_statutes and costly for de_statutes, but impossible for disease, where it would delete the dataset.

2. `de_statutes` is lexically inverted. The BM25 gate that removes keyword-solvable rows was set aggressively, and it removed not just the easy rows but every lexically normal one: the gold outscores its best negative on content-word overlap in 1 row out of 2,122. Real queries usually do share vocabulary with their relevant document, and training only on anti-lexical pairs risks teaching a retriever to distrust lexical overlap. uk_statutes shows the same pattern. If this matters for your mix, blend in rows generated the same way but rejected by that gate; they are held back and can be added on request.

3. Difficulty is not comparable across configs. BM25 ranks the gold first for 6-7% of statute rows but 46.5% of disease rows -- the disease query names the disease, so roughly half are keyword-solvable. Disease queries are also short templates (450 distinct templates; p50 50 characters) while statute queries are generated narratives (p50 ~1,485 characters). The statute sets carry a query_short field if you need the short shape.

4. Weak golds in `disease`. For ~6.7% of rows the judge found the gold does not answer its own query. These are not removed -- that is a source-data defect rather than a negatives defect, and the judge is not ground truth. Filter on your own criteria if it matters.

5. Synthetic queries. Statute queries are model-written fact situations, not real user queries. They are verified to be answerable by the gold and by nothing else in their row, but they are not drawn from a query log.

6. `letter_sibling` label collisions in `uk_statutes`. 6.1% of negatives carry the same section label as their gold (a section paired with a same-numbered Schedule paragraph). This is intentional -- identical-looking citation, different content is a good hard negative -- but it will look like a duplicate if you key on the citation.

Sources and licensing

• —de_statutes: gesetze-im-internet.de, German federal law.
• —uk_statutes: legislation.gov.uk UK Public General Acts, Open Government Licence v3.0. Note that ~58% of crawled Acts are metadata-only stubs (NumberOfProvisions="0") whose text was never digitised; those contribute nothing here.
• —disease: disease articles keyed to MONDO.

Queries and verification judgements are model-generated. Consult each upstream source for its own terms before redistribution.