Modelpublic

saiphanikrishna/domaintune-qwen2.5-3b-sec

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes

Model Card

DomainTune — Qwen2.5-3B fine-tuned on SEC 10-K filings

A 3B-parameter open model fine-tuned with QLoRA + DoRA + DPO on 500 real SEC 10-K annual reports to reliably extract structured financial data.

—Revenue accuracy: 50% → 84% (+34pp over zero-shot)
—Net income accuracy: 42% → 88% (+46pp)
—All-5-correct: 2% → 20% (10× improvement)
—Evaluated: n=50 held-out test set, 95% bootstrap CI (2,000 resamples)

Model Details

Property	Value
Base model	`Qwen/Qwen2.5-3B-Instruct`
Adapter	QLoRA r=16, α=16, DoRA=True
Target modules	q/k/v/o/gate/up/down proj
Trainable params	~30.9M (≈1% of base)
SFT training	200 steps · cosine LR 2e-4 · batch 16
DPO alignment	50 steps · β=0.1
Hardware	Google Colab T4 (free tier)
Training time	~90 minutes total

Task

Given an excerpt from a 10-K filing (income statement + Item 1A Risk Factors + Item 7 MD&A), extract five structured fields:

json

{
  "revenue_usd":     60922000000,
  "net_income_usd":  29760000000,
  "eps_diluted":     1.30,
  "top_risk_factor": "Competition in AI chip market from AMD, Intel, and custom silicon...",
  "mda_summary":     "Record revenue driven by Data Center segment growth of 217% YoY."
}

Key Finding — Unit-Scaling Bug

10-K income statements report figures with a qualifier: "amounts in millions of USD unless otherwise stated." The base model ignores this and outputs raw XBRL numbers (60922 instead of 60922000000). Injecting a [UNIT NOTE] token into every training example drove revenue accuracy from 24% → 86%.

Evaluation Results

Metric	Base (zero-shot)	SFT	DPO
JSON valid	96% [90–100]	100% [100–100]	98% [94–100]
Revenue within 1%	50% [36–64]	86% [76–94]	84% [74–94]
Net income within 1%	42% [28–56]	86% [76–96]	88% [78–96]
EPS within $0.01	88% [78–96]	100% [100–100]	98% [94–100]
Top risk (ROUGE-1 ≥ 0.35)	54% [40–68]	50% [36–64]	62% [48–76]
MD&A (ROUGE-1 ≥ 0.35)	10% [2–20]	38% [24–52]	36% [24–50]
All 5 correct	2% [0–6]	16% [6–26]	20% [10–32]

n=50 · 95% CI · bootstrap n=2000 · ROUGE-1 ≥ 0.35 threshold for text fields

How to Use

This repo contains the LoRA adapter only. Load it on top of the base model:

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_id    = "saiphanikrishna/domaintune-qwen2.5-3b-sec"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base      = AutoModelForCausalLM.from_pretrained(
    base_model_id, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

SYSTEM = (
    "You are a financial filing analyst. Given an excerpt from a company's SEC 10-K filing, "
    "extract exactly the following five fields and return ONLY valid JSON:\n"
    '{"revenue_usd": <number>, "net_income_usd": <number>, "eps_diluted": <number>, '
    '"top_risk_factor": "<1-3 sentences>", "mda_summary": "<1 sentence>"}\n\n'
    "CRITICAL units: if the income statement says 'in millions' multiply by 1,000,000; "
    "'in thousands' multiply by 1,000. EPS is already in $/share."
)

def extract(filing_text: str, unit_note: str = "amounts in millions of USD") -> str:
    prompt = f"[UNIT NOTE] All amounts in this filing are {unit_note}. Multiply accordingly.\n\n{filing_text}"
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}]
    ids = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
    with torch.no_grad():
        out = model.generate(ids.to(model.device), max_new_tokens=256, temperature=0.0, do_sample=False)
    return tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)

Faster with Unsloth (recommended for inference)

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "saiphanikrishna/domaintune-qwen2.5-3b-sec",
    max_seq_length=4096,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Training Data

500 real SEC 10-K filings from EDGAR (2018–2024) across 5 sectors × 2 size tiers. Numeric labels sourced from XBRL via `edgartools`. Text labels (top risk factor, MD&A summary) drafted by Claude Haiku and reviewed manually.

Split	N	Purpose
Train	400	SFT fine-tuning
Val	50	DPO preference pair construction
Test	50	Held-out evaluation

Limitations

—Designed for US GAAP 10-K filings only. Not validated on 10-Q, 20-F, or IFRS reports.
—Test set is n=50. Confidence intervals are wide — interpret point estimates with caution.
—Bank/financial-sector revenue (NII vs total) remains a known failure mode.
—Text field scoring uses ROUGE-1 ≥ 0.35; LLM-as-judge evaluation shows higher accuracy.

Citation

bibtex

@misc{arumalla2025domaintune,
  title  = {DomainTune: Fine-tuning Qwen2.5-3B for SEC 10-K Structured Extraction},
  author = {Arumalla, Sai Phani Krishna},
  year   = {2025},
  url    = {https://github.com/Saiphanikrishna05/DomainTune}
}