bowang0911/sec-xbrl-hard-negatives

SEC XBRL hard negatives

~461k query / positive / hard-negative triplets built from the SEC's Financial Statement Data Sets (DERA), covering 2023Q1–2026Q1, in two document granularities.

from datasets import load_dataset
load_dataset("bowang0911/sec-xbrl-hard-negatives", "section")   # default
load_dataset("bowang0911/sec-xbrl-hard-negatives", "page")

section is the sharper contrast — its extra axes (concept, gaap_vs_custom, per_share) slide a window down the same statement, so the negative differs from the positive by a few lines. page drops those three, because once the retrieval unit is the whole statement "another window of the same page" is not a document anyone can retrieve. page also matches how FinanceBench and TAT-QA chunk their corpora, at the cost of a higher collision rate (see below).

Every document is a fragment of a real financial statement, rendered from XBRL presentation data. Every hard negative is the same rendering of a deliberately wrong fact — wrong company, wrong period, wrong line item, wrong segment. The point is instance discrimination: the negatives are topically identical to the positive and differ only in the attribute the query names.

QUERY
  What was ADAMS RESOURCES & ENERGY, INC.'s total current liabilities
  as of December 31, 2022?

POSITIVE
  ADAMS RESOURCES & ENERGY, INC. - Consolidated Balance Sheets
  As of December 31, 2022
  Consolidated
    Derivative liabilities: $330,000
    Current portion of finance lease obligations: $4,382,000
    Current portion of operating lease liabilities: $2,712,000
    Other current liabilities: $19,214,000
    Total current liabilities: $231,060,000          <- the answer
    Long-term debt: $24,375,000
    ...

HARD NEGATIVE  [concept]   same company, same date, window slid up the
                           statement -- "Total assets" instead
HARD NEGATIVE  [fiscal_year]  same company, same line, December 31 2021
HARD NEGATIVE  [entity]    a peer filer's balance sheet, same line, same date

Why

Retrievers trained on web-page mixes learn topical separation and never learn fine-grained instance discrimination, which shows up as right topic, wrong company. These triplets isolate exactly that: for any given negative, the positive is the only document that matches on all of company, period, statement and line item.

Schema

Statistics

Negatives by axis (share of that config's negatives)

Statement IS 202,664 · BS 150,932 · CF 90,816 · CI 12,363 · UN 4,108 · SI 108 Form 10-Q 319,441 · 10-K 130,191 · 10-Q/A 6,046 · 10-K/A 5,445 mean query 85 chars. page documents: p10 513 · p50 1,147 · p90 1,939 · p99 2,891 chars.

Verified on an 8% sample of `page`: the line answering the query is present in the positive 100.00% of the time, and no negative shares the positive's exact header — every negative is separable from its first three lines alone.

Construction notes

• —One renderer for positives and negatives. Both sides go through the same formatter, so nothing about the surface form distinguishes them. Mixing the filer's own string for positives with a generated one for negatives would be a shortcut cue rather than a retrieval signal.
• —Facts are dated by `num.ddate` + `num.qtrs`, never by `sub.fy`/`sub.fp`. Those describe the filing: a FY2024 10-K reports FY2022 revenue as a comparative and still carries fy=2024.
• —`qtrs` is not collapsed. A 10-Q carries the discrete quarter (qtrs=1) and the year-to-date window (qtrs=2/3) at the same ddate; treating them as one would silently equate "three months ended Sep 30" with "nine months ended Sep 30". It is an axis instead.
• —Dimensional facts are kept. Rows with a non-empty segments are the richest near-duplicate source — same company, period and concept, one slice different.
• —Redundant negatives are dropped. A negative sharing more than 60% of its value lines (Jaccard) with the positive or with an already-selected negative is discarded, and per-axis-group budgets stop window-sliding from crowding out the structural axes.
• —No negative equals its own row's positive. Verified 0.
• —No negative contains the line that answers the query. Two filers can report the same label at the same value ("Goodwill: $0"), which makes a distractor a correct answer that no margin mask can catch — it duplicates the one line that matters, not the document. Those negatives are dropped: 0.51% of page-scale rows lost at least one, ~three quarters of them on entity.

Known limitations — read before training

Some hard negatives are another row's positive — 15.9% of negatives in section (54.7% of rows), 48.7% in `page` (91.3% of rows). This is intrinsic: the corpus is a fixed set of filings, so one query's distractor is another query's answer. page is far worse because coarsening the retrieval unit shrinks the document space — there is roughly one canonical page per (company, statement, period), so an entity negative is the peer's own positive, whereas a 9-line window is one of many.

The corpus-level rate overstates the harm: a collision only bites if the row owning that positive lands in the same batch, which is ≈(B−1)/(N−1) per negative. At 461k rows and batch size 128 that is ~2.8% of batches for section and ~6.7% for page.

If your collate pools hard negatives batch-wide, dedupe pooled negatives against in-batch positives by exact string match before computing the loss. That takes both to zero, and for page it is not optional.

Do not rely on a margin mask to catch this. The usual mask = neg_score > pos_score + threshold fires on the mislabeled-gold signature. An identical string scores exactly equal to the positive, so equal > equal + 0.2 is False and it trains as a negative at full weight. Worse, the mask can suppress genuine near-duplicate negatives that beat the gold early in training — the highest-gradient examples you have.

If your sampler draws negatives with replacement, expected distinct negatives from k draws over a pool of n is n(1-(1-1/n)^k). At 4.9 negatives/query this loses roughly 3% of slots.

~0.33% of queries are duplicated (1,524 in section, 1,523 in page), an artifact of the build being run in two passes over the 13 quarters.

Contamination: these are training-split filings from 2023Q1–2026Q1. Check period overlap before evaluating on FinQA, FinanceBench or TAT-QA, whose corpora are also drawn from SEC filings.

Source and licence

Built from the SEC Financial Statement Data Sets, which are US government works in the public domain. Released as CC0-1.0. The SEC does not endorse this dataset.