Modelpublic

raffi-souren/llm-output-drift-financial-workflows

sourceHugging Facemitupdated 10mo agoView on Hugging Face

0likes

Model Card

LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows

Authors: Raffi Khatchadourian, Rolando Franco Venue: AI4F @ ACM ICAIF 2025 (Nov 15 in Singapore)

Paper: https://arxiv.org/abs/2511.07585 Code: https://github.com/ibm-client-engineering/output-drift-financial-llms

This repository is a Hugging Face landing page for the paper and its open-source implementation. It focuses on deterministic test harnesses, cross-provider validation, and risk-tiered deployment for financial LLM workflows (SEC 10-Ks, RAG over filings, JSON/SQL tasks).

🔑 Key finding

Well-engineered 7–8B models achieve 100% output consistency at T=0.0, while a 120B model reaches only 12.5% consistency, regardless of configuration.

Across 480 runs (5 models, 3 tasks, 2 temperatures, 3 concurrency levels), we show an inverse relationship between model size and determinism and map this to regulatory requirements (FSB, BIS, CFTC).

📊 Model tier classification

Tier	Models	Consistency @ T=0.0	Status	Recommended use
1	Granite-3-8B, Qwen2.5-7B	100%	✅ Production-ready	All regulated tasks
2	Llama-3.3-70B, Mistral-Medium-2505	56–100%	⚠️ Task-specific	SQL / structured only
3	GPT-OSS-120B	12.5%	❌ Non-compliant	Not for compliance

n = 480 runs (16 per condition), 95% Wilson CIs, p < 0.0001 (Fisher’s exact).

🎯 Why this matters

Financial institutions face a “verification tax”: human review erodes AI productivity gains when outputs are nondeterministic.

This framework shows:

—Audit-ready determinism is achievable with the right model + decoding setup.
—Cross-provider consistency: behavior transfers between local (Ollama) and cloud (IBM watsonx.ai).
—Task-specific drift: SQL and structured summaries remain stable even at T=0.2; RAG is far more sensitive.