nate149/rural-mortgage-bot-api

git clone https://github.com/YOUR-USERNAME/rural-mortgage-bot.git

cd rural-mortgage-bot

python -m venv venv

source venv/Scripts/activate    # Windows Git Bash

\# OR:  source venv/bin/activate  # Mac/Linux

pip install -r requirements.txt



2\. Get API keys



OpenAI: https://platform.openai.com/api-keys

Pinecone: https://app.pinecone.io

Census (free): https://api.census.gov/data/key\_signup.html



3\. Create .env in project root



OPENAI\_API\_KEY=sk-proj-...

PINECONE\_API\_KEY=pcsk\_...

PINECONE\_INDEX\_NAME=rural-mortgage

CENSUS\_API\_KEY=...



4\. Download reference datastreamlit run app.py



mkdir -p data/reference

curl -L -o data/reference/cfpb\_rural\_counties.csv \\

  "https://files.consumerfinance.gov/f/documents/cfpb\_rural-or-underserved-counties\_2024.csv"



5\. Build the Knowledge Base (first time only)



python download\_pdfs.py

python ingest.py

python web\_ingest.py



6\. Run



streamlit run app.py


Data Sources

The bot draws from three categories of data: bundled PDFs, reference CSVs, and live APIs.

### 📄 PDF handbooks (`data/pdfs/`)

USDA, FHA, and VA program handbooks plus rural reference materials. Total ~30MB.

**USDA Section 502 Direct (Handbook 3550-1)**
- `3550-1chapter01.pdf` — Overview
- `3550-1chapter02.pdf` — Record Retention
- `3550-1chapter03.pdf` — Lender Approval
- `3550-1chapter04.pdf` — Borrower Eligibility
- `3550-1chapter05.pdf` — Property Requirements
- `3550-1chapter06.pdf` — Loan Purposes
- `3550-1chapter07.pdf` — Loan Terms
- `3550-1chapter08.pdf` — Application Processing
- `3550-1chapter09.pdf` — Loan Approval
- `3550-1chapter10.pdf` — Loan Closing

**USDA Section 502 Guaranteed (Handbook 3555-1)**
- `3555-1chapter01.pdf` — Overview
- `3555-1chapter03.pdf` — Lender Approval
- `3555-1chapter04.pdf` — Lender Responsibilities
- `3555-1chapter05.pdf` — Origination
- `3555-1chapter06.pdf` — Loan Purposes
- `3555-1chapter07.pdf` — Loan Terms
- `3555-1chapter08.pdf` — Applicant Characteristics
- `3555-1chapter09.pdf` — Income Analysis
- `3555-1chapter10.pdf` — Credit Analysis
- `3555-1chapter11.pdf` — Ratio Analysis
- `3555-1chapter15.pdf` — Submitting the Application
- `3555-1chapter16.pdf` — Closing

**FHA Single Family Housing**
- `40001-hsgh-Update-17.pdf` — FHA Single Family Housing Policy Handbook 4000.1

**VA**
- `VA_Buyers_Guide.pdf` — VA Home Loan Buyer's Guide

**USDA Reference**
- `RD-DirectLimitMap.pdf` — USDA Direct Loan income limit map
- `RD-SFH-ManufacturedHousing1.pdf` — USDA manufactured housing requirements

**CFPB Rural/Underserved (2026)**
- `cfpb_rural-list_2026.pdf`
- `cfpb_rural-underserved-list_2026.pdf`
- `cfpb_underserved-list_2026.pdf`

### 📊 Reference CSVs (`data/reference/`)
- `cfpb_rural_counties.csv` — CFPB rural/underserved county designations (machine-readable)

### 🌐 Ingested web pages
See `data/.ingested_urls.txt` for the list of program web pages scraped by `web_ingest.py`.

### 🔌 Live APIs (no local data needed)
| Source | Use |
|---|---|
| **Pinecone** | Vector store for RAG search over PDFs and web content |
| **OpenAI** | Embeddings (`text-embedding-3-large`) and chat (`gpt-4o`) |
| **Census Bureau Geocoder** | Address → county FIPS, lat/lon (free, no key) |
| **Census Bureau ACS** | Place population estimates (free, requires API key) |
| **CFPB FFIEC HMDA Data Browser** | Lender activity by state/county/loan type (free, no key) |

### Re-ingestion

To rebuild the knowledge base from scratch: