akshit1229/credit-statement-intelligence

Credit Statement Intelligence Engine

A production-grade AI system for credit statement intelligence using RAG (Retrieval-Augmented Generation) and intent-based query routing.

🎯 Features

• —PDF Ingestion: Parse multiple credit card statement PDFs with different formats
• —Smart Data Extraction: Extract normalized transactions and statement metadata
• —Vector Store: Semantic search using ChromaDB and sentence transformers
• —Intent Classification: Route queries using 5 intent types (list, aggregate, compare, topn, generalexplanation)
• —Evidence-Backed Answers: No hallucinations - all responses include source evidence
• —Analytics & Reconciliation: Spend analysis, category breakdown, and data validation
• —REST API: FastAPI with interactive documentation

📋 System Requirements

• —Python 3.11+
• —2GB RAM minimum (4GB recommended for vector embeddings)
• —Groq API key (for LLM-based intent classification)

🚀 Quick Start

1. Installation

# Clone the repository
git clone <repository-url>
cd credit-statement-intelligence

# Create virtual environment
python -m venv .venv
.venv\Scripts\activate  # Windows
# source .venv/bin/activate  # Linux/Mac

# Install dependencies
pip install -r requirements.txt

2. Configuration

Create a .env file in the project root:

GROQ_API_KEY=your_groq_api_key_here

3. Run the Server

python main.py

The server will start on http://localhost:8000

• —API Docs: http://localhost:8000/docs
• —ReDoc: http://localhost:8000/redoc

📡 API Endpoints

💡 Usage Examples

Upload a PDF Statement

curl -X POST "http://localhost:8000/ingest" \
  -F "file=@statement.pdf" \
  -F "statement_year=2023"

Ask Questions

# Aggregate query
curl -X POST "http://localhost:8000/query" \
  -H "Content-Type: application/json" \
  -d '{"question": "How much did I spend on dining?"}'

# Top N query
curl -X POST "http://localhost:8000/query" \
  -H "Content-Type: application/json" \
  -d '{"question": "What are my top 5 expenses?"}'

# RAG query (general explanation)
curl -X POST "http://localhost:8000/query" \
  -H "Content-Type: application/json" \
  -d '{"question": "What fees are mentioned in my statement?"}'

Get Analytics

# Summary with category breakdown, top merchants, time analysis
curl "http://localhost:8000/summary"

# Reconciliation report
curl "http://localhost:8000/reconciliation"

# Export to CSV
curl "http://localhost:8000/export/csv" -o transactions.csv

🏗️ Architecture

credit-statement-intelligence/
├── app/
│   ├── ingestion/          # PDF parsing and data extraction
│   │   ├── parsers/        # PDF parser implementation
│   │   ├── normalizers/    # Transaction normalization
│   │   └── storage.py      # DuckDB storage layer
│   ├── vector_store/       # ChromaDB vector store
│   ├── query_router/       # Intent classification & query execution
│   ├── analytics/          # Spending insights
│   ├── reconciliation/     # Data validation
│   └── api/                # FastAPI routes
├── data/
│   ├── raw/                # Uploaded PDFs
│   ├── processed/          # Processed data
│   ├── vector_db/          # ChromaDB persistence
│   └── statements.db       # DuckDB database
├── outputs/                # CSV exports and reports
└── tests/                  # Test suite

Technology Stack

• —Backend: FastAPI, Uvicorn
• —PDF Parsing: pdfplumber
• —Database: DuckDB (analytical queries)
• —Vector Store: ChromaDB + sentence-transformers
• —LLM: Groq (llama-3.3-70b-versatile)
• —Data Processing: pandas, numpy
• —Testing: pytest

🧪 Testing

Run Unit Tests

pytest tests/test_query_router.py -v

Test Results: All 11 tests passed ✅

• —Intent classification (aggregate, list, topn, compare, generalexplanation)
• —Query plan structure validation
• —Multiple query type tests

Manual API Testing

1. 1.Start the server: python main.py
2. 2.Open http://localhost:8000/docs
3. 3.Test endpoints via Swagger UI

🔍 Intent Classification

The system classifies user questions into 5 intents:

1. 1.list: "Show me all dining transactions"
2. 2.aggregate: "How much did I spend on groceries?"
3. 3.compare: "Compare January vs February spending"
4. 4.top_n: "What are my top 5 expenses?"
5. 5.general_explanation: "What is the annual fee?" (uses RAG)

Each query generates a structured query plan:

{
  "intent": "aggregate",
  "query_plan": {
    "filters": {
      "category": "dining",
      "txn_type": "debit"
    },
    "operation": "sum",
    "parameters": {}
  },
  "requires_rag": false,
  "confidence": 0.95
}

📊 Analytics Features

• —Category Breakdown: Spending by category (dining, groceries, transportation, etc.)
• —Top Merchants: Highest spending merchants
• —Time Analysis: Monthly and weekly spending patterns
• —Unusual Transactions: Flagged high-value or duplicate transactions
• —Spending Patterns: Day-of-week and month-period analysis

🔒 Data Safety

• —No Hallucinations: All answers include evidence from actual data
• —Source Attribution: Retrieved document chunks with page references
• —Validation: Reconciliation against statement totals
• —Error Handling: Proper validation when no data is available

🎓 Design Decisions

Why DuckDB?

• —Analytical query performance for aggregations
• —SQL interface for complex filtering
• —Efficient columnar storage

Why ChromaDB?

• —Easy Python integration
• —Good performance for moderate datasets
• —Built-in persistence

Why Groq?

• —Fast inference for intent classification
• —Good structured output support
• —Cost-effective for production use

📝 Assumptions

1. 1.PDF Format: Statements contain tabular transaction data
2. 2.Date Format: Transactions have parseable dates (DD/MM/YYYY or similar)
3. 3.Amount Format: Amounts are in decimal format with "Cr" suffix for credits
4. 4.Statement Year: May need to be provided if not in PDF
5. 5.Category Matching: Based on keyword matching in merchant names

🐛 Known Issues / Future Improvements

• —[ ] Support for more PDF formats (OCR for scanned PDFs)
• —[ ] Multi-currency support
• —[ ] User authentication and multi-tenant support
• —[ ] Export to multiple formats (Excel, JSON)
• —[ ] Advanced duplicate detection
• —[ ] Machine learning for category classification

📖 Sample Outputs

After ingestion, the system generates:

• —transactions.csv: All extracted transactions
• —summary.json: Analytics summary
• —reconciliation_report.json: Validation report

See the outputs/ directory for examples.

🤝 Contributing

This is an assignment project. For production use, consider:

• —Adding authentication/authorization
• —Implementing rate limiting
• —Adding comprehensive logging
• —Database connection pooling
• —Caching for frequent queries

📄 License

This is an educational project created as an assignment.

🙏 Acknowledgments

Built using:

• —FastAPI framework
• —ChromaDB vector database
• —Groq LLM API
• —pdfplumber library