Modelpublic

Fernandosr85/consumer-finance-complaints-cfpb-adapter

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes

Model Card

banner

Consumer Finance Complaints — CFPB Taxonomy Adapter

LoRA adapter for hierarchical classification of CFPB consumer complaint narratives, fine-tuned on Mixtral-8x7B-Instruct-v0.1 via Adaption's AutoScientist platform.

Given a complaint narrative, the adapter emits a three-level label from the CFPB taxonomy: Product → Issue → Sub-issue, where each level is constrained to be valid under its parent.

Task

This is closed-label classification, not open generation. The output space is a fixed taxonomy, not free text:

Level	Cardinality
Products	11
Issues	86
Sub-issues	209
Valid (Product, Issue, Sub-issue) triples	393

The unconstrained space would be 11 × 86 × 209 ≈ 197,714 combinations. The taxonomy's hierarchy reduces that to 393 valid triples — a 503× reduction. A correct output is one of those 393; anything else is a label hallucination.

Output format is three lines, verbatim labels, no explanation:

Product: <label>
Issue: <label>
Sub-issue: <label>

Evaluation results

Training win rates (Adaption internal)

Model	Win Rate (in-domain)
Base (`Mixtral-8x7B-Instruct-v0.1`)	17%
Adapted (`consumer_finance_complaints`)	83%

A +66 point improvement on the in-domain evaluation.

Cross-domain tradeoff (catastrophic forgetting). The adapter specializes at the cost of general capability. On the Legal reference category, win rate drops from 64% (base) → 36% (adapted). This is expected for a task-specific adapter and is reported for transparency, not hidden.

Held-out evaluation (this is the comparable number)

The win rate above is Adaption's internal metric. The numbers below are measured on a temporal holdout the adapter never saw during training (742 rows, July 2025), scored with the same parser and the same 393-triple registry used to evaluate every other method.

Method	Triple exact-match	Hallucination rate
Majority-triple baseline	16.2%	0%
Claude zero-shot (Sonnet)	20.3%	21.6%
TF-IDF + SGD (300k train)	37.6%	0%
This adapter	`[PENDING — run holdout eval]`	`[PENDING]`

The adapter's value proposition is not only accuracy but label validity: a closed-label task where the base model hallucinates 21.6% of the time, and where the adapter — having seen the 393 valid triples during training — should drive that toward zero. Fill this table before publishing.

Train/eval metrics

Metric	Value
Loss curve	monotone decrease, plateau after ~18 steps
Train vs validation	tracked closely, no overfit divergence
LR scheduler	cosine (warmup 0.05)
Gradient norm	initial spike, stable thereafter

Validation loss tracked train loss without divergence across all 5 epochs, confirming generalization rather than memorization — important given the small (393) output space.

Model details

Field	Value
Base model	`mistralai/Mixtral-8x7B-Instruct-v0.1` (46.7B)
Trained model name	`adaption_consumer_finance_complaints`
Training method	Supervised Fine-Tuning (SFT) + LoRA
LoRA rank (r)	64
LoRA alpha	128
LoRA dropout	0
Trainable modules	all-linear
Epochs	5
Batch size	max
Learning rate	1e-4 (cosine, 0.5 cycles)
Warmup ratio	0.05
Weight decay	0.05
Max grad norm	1.0
Train on inputs	false
Data format	Chat (instruction/response)

Inference note. data_format was chat and train_on_inputs was false. To reproduce results, wrap the narrative as a user turn via the tokenizer's chat template — do not feed raw text. Mismatched formatting measures format error, not model quality.

Training dataset

[Fernandosr85/adaption-consumer-finance-complaints](https://huggingface.co/datasets/Fernandosr85/adaption-consumer-finance-complaints)

12,346 prompt/completion pairs, stratified by (Product, Issue) pair. Built from the full CFPB complaint dump with a deliberate methodology:

Step	Detail
Taxonomy vintage	Fixed at the 2023-08-25 consolidation cutoff (11 current Products)
Deduplication	52.1% of post-2023 narratives are exact duplicates after normalization; removed via exact + template (MinHash-LSH) dedup
Class rebalance	Credit reporting falls from 72.6% → 46.2% of the corpus after dedup
Split	Temporal holdout (train < 2025-07, eval = July 2025)
Registry	Derived from train only; eval never contributes to the label set
Hard negatives	Boosted from a measured product-confusion matrix, not intuition

Methodological note on the eval window. CFPB narrative publication collapses from August 2025 (intake continues, publication does not) and is effectively zero by 2026. Any naive temporal holdout on recent CFPB data is contaminated by this. The July 2025 window is the last month at the stable publication plateau.

Known limitations

—Completion regeneration. Adaptive Data regenerates completions on every run; a freeform Blueprint cannot enforce membership in a closed label set. This adapter was trained on the original (verbatim) completions, not the platform-enhanced ones, which contained retired labels (e.g. Bank account or service, removed by the CFPB in August 2023). Downstream users of the platform on closed-label tasks should map the original completion column.
—Single-month eval. Held-out evaluation is one month (July 2025) for the reasons above. A replication on June 2025 would strengthen the temporal-robustness claim.
—Domain specialization. See the Legal cross-domain drop above. This adapter is not intended for general-purpose use.

Model repositories

Platform	Link
HuggingFace Model	Fernandosr85/consumer-finance-complaints-cfpb-adapter
HuggingFace Dataset	Fernandosr85/adaption-consumer-finance-complaints

Credits

—Fine-tuning platform: Adaption — AutoScientist & Adaptive Data
—Challenge: AutoScientist Challenge 2026 — Personal Finance category
—Data source: CFPB Consumer Complaint Database (public)
—Author: Fernando Rodrigues · Kaggle: fernandosr85 · HuggingFace: Fernandosr85

Disclaimer

Experimental research artifact submitted to the AutoScientist Challenge 2026 (Personal Finance category). Classification outputs are automated and require review before any operational use.