FatimahM24/Insurance-Claim-Classification-Fraud-Detection-and-Action-Recommendation-System

Insurance Claim Fraud Detection & Action Recommendation System

An explainable machine-learning system that screens vehicle-insurance claims for fraud, assigns each claim a calibrated risk tier, and generates a human-readable investigation brief with a recommended next action. Built as the capstone project for the Samsung Innovation Campus AI Course.

Scope note. This is a proof-of-concept validated on a historical benchmark dataset (1994–1996). It demonstrates a leakage-safe, calibration-focused methodology rather than a system ready for present-day deployment; real-world use would require retraining on current claims data.

Table of Contents

• —Overview
• —Methodology
• —Results
• —Repository Structure
• —Installation & Usage
• —Team

Overview

Manual insurance claim processing is slow, resource-intensive, and error-prone, often allowing fraudulent claims through while delaying legitimate payouts. This project builds a decision-support system that:

1. 1.Classifies each claim as fraudulent or legitimate using gradient-boosted decision trees.
2. 2.Calibrates the predicted probabilities so a "20% risk" score genuinely means ~20%.
3. 3.Assigns risk tiers (Low / Medium / High) from the calibrated probabilities.
4. 4.Explains each decision with TreeSHAP feature attributions.
5. 5.Recommends an action (approve / manual review / investigate) via a generative reasoning model, grounded in the SHAP evidence and subject to a human-in-the-loop check.

The guiding principles throughout are leakage-safe evaluation, honest handling of class imbalance, and trustworthy (calibrated) probabilities. ---

Methodology

Primary metric: PR-AUC. Reported alongside Recall, Precision, Macro-F1, MCC, and calibration (Brier / reliability).

Results

Note on Model Comparison & Evaluation: Untuned baseline models (Logistic Regression, Balanced Random Forest) are evaluated directly on the left-out validation set. Candidate models >(XGBoost, LightGBM) are tuned and compared using 5-fold cross-validation on the training set. These scores reflect different evaluation setups and are reported in separate columns to ensure fair, transparent comparisons.

Final model: XGBoost + class_weight · Test-set PR-AUC: 0.195

Key figures are in `results/figures/`.

Repository Structure

insurance-fraud-detection/
│
├── README.md
├── requirements.txt
├── .gitignore                 # ignores notebooks/prompts/, notebooks/results/, notebooks/rubric/
│                               #   (incl. brief_scores.csv), __pycache__/, .ipynb_checkpoints/
│
├── notebooks/                 # run order
│   ├── 01_eda_and_statistics.ipynb
│   ├── 02_preprocessing_and_splitting.ipynb
│   ├── 03_baseline.ipynb
│   ├── 04_modelling_and_selection.ipynb        # Experiment + Optuna → select on VALIDATION (no test set here)
│   ├── 05_calibration_and_thresholds.ipynb     # calibrate + tune threshold/tiers on VALIDATION
│   ├── 06_final_test_evaluation.ipynb          # LOCKED TEST SET — run once, after model+calibration+thresholds frozen
│   ├── 07_shap_explainability.ipynb
│   ├── 08_qwen_briefs_and_rubric.ipynb         # runs locally (Ollama) — generates + scores GenAI briefs
│   ├── shap_claim_briefs.json                  # GIT-IGNORED — Notebook 07's handoff, copied down from Drive
│   ├── prompts/                                # GIT-IGNORED — prompt text, never committed
│   │   └── brief_prompt.txt
│   ├── results/                                # GIT-IGNORED — Notebook 08's local output
│   │   └── generated_briefs.json
│   └── rubric/                                 # GIT-IGNORED — human-scored rubric CSV, local only
│       └── brief_scores.csv
│
├── app.py                     # Gradio demo — serves the team's precomputed, rubric-passed results
├── model_pipeline.py          # loads demo_data.json; powers app.py's predictions/metrics
├── demo_data.json             # 46 real SHAP+brief samples (Low/Medium/High) used by the live demo
│
├── assets/                    # logos used by app.py's header
│   ├── innovexa_header_mark.png
│   └── innovexa_logo.png
│
├── data/
│   └── README.md              # dataset source + CC0 license
│
├── results/                   # small final outputs committed for graders
│   ├── metrics.json
│   └── figures/
│
└── report/
    └── final_report.docx

Installation & Usage

git clone https://github.com/leen449/insurance-fraud-detection.git
cd insurance-fraud-detection
pip install -r requirements.txt

Environment: developed in Google Colab. Key libraries: scikit-learn, xgboost, lightgbm, imbalanced-learn, optuna, shap. See requirements.txt for versions.

Google Drive Setup (Notebooks 01–07)

Notebooks 01–07 run in Colab against a shared Drive folder, mounted as PROJECT_ROOT:

InsuranceFraudProject/                 # My Drive - PROJECT_ROOT for Notebooks 01-07
│
├── data/
│   ├── raw/
│   │   └── fraud_oracle.csv           # ADD MANUALLY - the only file you place by hand
│   └── processed/                     # auto-written by Notebook 02
│       ├── cleaned.parquet
│       ├── train.parquet
│       ├── val.parquet
│       ├── test.parquet
│       ├── train_temporal.parquet
│       ├── test_temporal.parquet
│       └── split_manifest.json
│
├── genai/                             # manual backup copy - no notebook reads/writes this folder
│   └── brief_prompt.txt
│
├── models/                            # auto-written by Notebooks 02, 04, 05
│   ├── preprocessing_pipeline.joblib
│   ├── best_model.joblib
│   ├── best_model_config.json
│   ├── calibrated_model.joblib
│   └── risk_config.json
│
├── results/
│   ├── figures/                       # auto-written by Notebooks 01, 03-07
│   │   ├── eda/
│   │   ├── baseline/
│   │   ├── modelling/
│   │   ├── calibration/
│   │   ├── test/
│   │   └── shap/
│   ├── experiments/                   # manual - not produced by any notebook
│   ├── experiments.csv                # auto-appended by Notebooks 03, 04, 05, 06
│   ├── fairness_audit_test.json       # auto-written by Notebook 06
│   ├── generated_briefs.json          # manual backup - Notebook 08 writes this locally, not to Drive
│   ├── metrics.json                   # auto-written by Notebook 06
│   └── shap_claim_briefs.json         # auto-written by Notebook 07 - see handoff note below
│
└── rubric/                            # manual backup - Notebook 08 writes this locally, not to Drive
    └── brief_scores.csv

Setting it up: create a folder named exactly InsuranceFraudProject in My Drive, then add data/raw/fraud_oracle.csv — that's the only file you place there by hand. Everything else under data/, models/, and results/ is generated automatically by running Notebooks 01→07 in order, in Colab, with the Drive mounted; each notebook reads what the previous one wrote and creates its own output folders as it goes.

Handoff to Notebook 08 (which runs locally, not in Colab): Notebook 07 writes results/shap_claim_briefs.json to Drive. Copy that one file down to notebooks/shap_claim_briefs.json on your machine before running Notebook 08 — that's the only Drive-to-local step in the whole pipeline; Notebook 08 needs no Drive access of its own, and writes its own outputs locally to notebooks/{prompts,results,rubric}/ (see that notebook's own setup notes).

`genai/`, `experiments/`, `generated_briefs.json`, and `rubric/brief_scores.csv` in the tree above are manually-kept backups, not something any Colab notebook produces — safe to ignore when setting up from scratch.

Team

Innovexa — Team 4 · Samsung Innovation Campus AI Course

References

• —shivamb. Vehicle Claim Fraud Detection. Kaggle. https://www.kaggle.com/datasets/shivamb/vehicle-claim-fraud-detection.
• —Komsrimorakot, P., & Siriborvornratanakul, T. (2025). Enhancing fraud detection in imbalanced motor insurance datasets using CP-SMOTE and Random Under-Sampling. Journal of Big Data, 12, 172.
• —S. Subudhi and S. Panigrahi, “Detection of automobile insurance fraud using feature selection and data mining techniques,” International Journal of Rough Sets and Data Analysis, vol. 5, no. 3, pp. 1–20, Jul. 2018, doi: 10.4018/IJRSDA.2018070101.