miscusi/adaption-market-analysis-sec

Market Analysis & News Instruction Dataset (SEC XBRL-grounded)

Instruction-tuning data for financial analysis — fundamentals, growth and ratio arithmetic, trend and risk reading, filing navigation and comparability caveats — built from real XBRL facts, with every stated figure independently re-derived.

Built for the Adaption Labs AutoScientist Challenge Part 2, Market Analysis & News track.

What is in it

Three design decisions, and why

Concise answers, and we tested whether that was right. Median response is 56 words. Pairwise judges are known to reward length, and a documented entry in this same challenge trained on ~780-word chain-of-thought and lost to its own base model twice. We tested the trade-off directly on the HR dataset card: training on 5.7x longer completions produced a model indistinguishable from the concise one, so concision costs nothing measurable and is far cheaper to serve.

Split by company, not by row. A pool of 507 companies was reserved before generation and excluded from training entirely; the 567 evaluation rows are drawn from 276 of them (verified: 0 companies overlap training). A row-level split would let the same company appear on both sides and report memorisation as generalisation.

Breadth over depth. 18 task families rather than one schema repeated. The target is competence across the domain, including tasks phrased in ways this dataset does not contain.

Verification

3,393 numeric claims across both datasets were independently re-derived from source and every one matches the figure stated in the response.

The verifier (verify.py, included in this repo) does not import the generator's arithmetic. It parses each question for its inputs, recomputes the answer from scratch — for the market set, from the source XBRL facts — and compares against the figure the stored response states. Sharing a helper would let a wrong formula agree with itself.

Adaption platform quality grade

Graded by Adaption's own data-quality evaluation (dataset bc48eb4e-1de9-4187-8094-d35f5304f8c8, sampled on 100 rows):

Improvement: +24.3%. All numbers, including the per-metric percentiles, are exactly as returned by the platform API (eval/adaption_grade_*.json in the build repo); where a before/after percentile repeats, that repetition is the platform's own coarse bucketing, not a transcription error.

Format

{"id": "...", "task_family": "...", "instruction": "...", "response": "...", "split": "train"}

Recommended system prompt:

You are a financial analyst. Answer with the figures that matter, state the arithmetic you used, and flag when a comparison is misleading. Be concise.

Licence and attribution

Released under CC-BY-4.0.

Financial figures are taken from XBRL company facts filed with the U.S. Securities and Exchange Commission and retrieved through the SEC's public API. U.S. government works are not subject to copyright. Figures are as filed and may have been restated by the filer since retrieval.

Source: SEC EDGAR XBRL company facts. (Note: sec.gov returns HTTP 403 to clients without a declared User-Agent — automated link checkers will flag it; it opens normally in a browser, per the SEC's fair-access policy.)

Mirrors and companion artifacts

The challenge requires the dataset and the weights on Hugging Face and Kaggle. All four artifacts for this track, plus the public demo:

Why trust these numbers

• —Every numeric claim in every response is re-derived from source by verify.py (included in this repo), which shares no arithmetic with the generator.
• —All model evaluations for this track are blinded and judged in both orderings: a verdict that does not survive swapping the answers is recorded as a tie, never resolved in our favour.
• —The demo publishes every evaluation prompt with all models' answers — including the ones we lose.

Citation

@misc{adaption_market_analysis_sec_2026,
  author = {Adia-Nimuwa, Usi},
  title  = {adaption-market-analysis-sec: AutoScientist Challenge Part 2, Market Analysis & News track},
  year   = {2026},
  url    = {https://huggingface.co/datasets/miscusi/adaption-market-analysis-sec}
}

Limitations

• —Responses are generated from structured records by template, then verified. They are factually grounded and stylistically consistent, which also means they are stylistically narrow — this set is designed to be mixed with general instruction data, not trained on alone.
• —Figures are as filed at retrieval time. Filers restate, and fiscal-year labels do not imply aligned periods across companies. Nothing here is investment advice.
• —English only.