Aigenthix/Graph_RAG7

1# Project Structure 📁2 3Complete overview of the Graph RAG Chatbot project files and their purposes.4 5```6graph-rag-chatbot/7├── 📄 Core Application Files8│   ├── app.py                      # Main Flask application (500+ lines)9│   ├── requirements.txt             # Python dependencies10│   └── .env.example                 # Environment variables template11│12├── 🐳 Docker & Deployment13│   ├── Dockerfile                   # Docker image definition14│   ├── docker-compose.yml           # Docker Compose configuration15│   ├── .dockerignore               # Files to exclude from Docker build16│   └── deploy.sh                    # Automated deployment script (Linux/Mac)17│   └── deploy.bat                   # Automated deployment script (Windows)18│19├── 📚 Documentation20│   ├── README.md                    # Complete documentation21│   ├── QUICKSTART.md               # 5-minute quick start guide22│   ├── TESTING.md                  # Comprehensive testing guide23│   ├── space_config.md             # HF Spaces deployment guide24│   └── PROJECT_STRUCTURE.md        # This file25│26├── 🎨 Frontend27│   └── templates/28│       └── index.html              # Complete responsive UI (HTML + CSS + JS)29│30├── 📦 Data Storage (created at runtime)31│   ├── data/32│   │   ├── uploads/                # Uploaded documents stored here33│   │   │   └── .gitkeep34│   │   └── graph_data/             # Knowledge graphs (PNG images)35│   │       └── .gitkeep36│37├── 🔧 Configuration38│   └── .gitignore                  # Git ignore rules39│40└── 📋 Optional Files (for your reference)41    ├── LICENSE                     # MIT License (optional)42    └── CONTRIBUTING.md             # Contribution guidelines (optional)43```44 45---46 47## File Details48 49### Core Application (`app.py`)50 51**Size**: ~550 lines52**Language**: Python 3.8+53**Dependencies**: Flask, Groq, SentenceTransformers, NetworkX54 55**Key Components**:56 571. **Flask Setup** (lines 1-50)58   - Initialize Flask app59   - Configure CORS60   - Set up upload folder61   - Initialize models62 632. **Document Processing** (lines 51-150)64   - `DocumentProcessor` class65   - Text extraction (PDF, CSV, TXT)66   - Text chunking with LangChain67 683. **Knowledge Graph Building** (lines 151-220)69   - `GraphBuilder` class70   - Create nodes and edges71   - Generate NetworkX graph72   - Visualize with Matplotlib73 744. **API Endpoints** (lines 221-450)75   - `GET /` - Serve UI76   - `GET /api/documents` - List documents77   - `POST /api/upload` - Upload files78   - `POST /api/query` - RAG queries79   - `GET /graph-image/<filename>` - Get graph PNG80   - `DELETE /api/delete/<filename>` - Delete document81 825. **Async Processing** (lines 451-550)83   - Background thread processing84   - Progress tracking85   - Error handling86 87### Frontend (`templates/index.html`)88 89**Size**: ~700 lines90**Language**: HTML + CSS + JavaScript91**No external build step required**92 93**Sections**:94 951. **Styling** (lines 1-350)96   - Modern gradient design97   - Responsive grid layout98   - Dark mode ready99   - Animations and transitions100 1012. **HTML Structure** (lines 351-500)102   - Upload zone103   - Document list104   - Chat interface105   - Graph viewer106   - Tabbed interface107 1083. **JavaScript** (lines 501-700)109   - File upload handling110   - Real-time document refresh111   - Chat message display112   - Graph visualization113   - API communication114 115### Configuration Files116 117#### `requirements.txt`118```119Flask==2.3.3                 # Web framework120Flask-CORS==4.0.0            # CORS support121python-dotenv==1.0.0         # .env loading122sentence-transformers==2.2.2 # Embeddings123groq==0.4.1                  # Groq API124PyPDF2==3.0.1                # PDF parsing125pandas==2.0.3                # Data handling126langchain==0.0.283           # Text processing127networkx==3.1                # Graph algorithms128matplotlib==3.7.2            # Graph visualization129numpy==1.24.3                # Numerical computing130torch==2.0.1                 # ML framework131```132 133#### `Dockerfile`134- Base: `python:3.11-slim` (compact, secure)135- Installs: gcc, g++ for C dependencies136- Installs: Python packages from requirements.txt137- Exposes: Port 7860138- CMD: Run Flask app139 140#### `docker-compose.yml`141- Service: `graph-rag`142- Port mapping: 7860:7860143- Environment: GROQ_API_KEY, PORT144- Volumes: ./data for persistence145- Health check: HTTP 200 on /146- Restart policy: unless-stopped147 148### Environment Variables (`.env`)149 150```env151GROQ_API_KEY=your_groq_api_key_here    # Required: LLM API access152PORT=7860                              # Optional: Application port153FLASK_ENV=production                   # Optional: production/development154```155 156**Never commit .env file!** Use `.env.example` as template.157 158### Data Storage159 160#### `data/uploads/`161- **Purpose**: Store uploaded documents162- **Contents**: PDF, CSV, TXT files163- **Persistence**: Survives container restarts164- **Size Limit**: 50MB per file165 166#### `data/graph_data/`167- **Purpose**: Store generated graph images168- **Format**: PNG files (DPI: 150)169- **Naming**: `{filename}_graph.png`170- **Size**: ~50-200KB per graph171 172---173 174## Technology Stack 🛠️175 176### Backend177- **Framework**: Flask (lightweight, easy to deploy)178- **API**: RESTful with JSON179- **Language**: Python 3.8+180- **LLM**: Groq Mixtral 8x7b181- **Embeddings**: SentenceTransformers (all-MiniLM-L6-v2)182- **Graphs**: NetworkX (algorithms, visualization)183 184### Frontend185- **Language**: HTML5 + CSS3 + Vanilla JavaScript186- **No frameworks**: Zero dependencies (lighter bundle)187- **Features**: Drag-and-drop, real-time updates, responsive design188- **Charts**: Native SVG visualization189 190### Infrastructure191- **Containerization**: Docker (Alpine-based)192- **Orchestration**: Docker Compose193- **Deployment**: HF Spaces, AWS, GCP, Azure194- **Storage**: Ephemeral (configurable)195 196---197 198## Data Flow 📊199 200### Upload Flow201```202User Upload203    ↓204Browser → POST /api/upload205    ↓206Flask receive file → Save to disk207    ↓208Queue async thread209    ↓210Return 200 OK (immediately)211    ↓212Background: Extract text213    ↓214Background: Chunk text215    ↓216Background: Build graph217    ↓218Background: Generate embeddings219    ↓220Frontend polls GET /api/documents221    ↓222Document shows "ready" status223    ↓224Graph image available225```226 227### Query Flow228```229User Query230    ↓231Browser → POST /api/query232    ↓233Embed query text234    ↓235Calculate cosine similarity with chunks236    ↓237Select top 3 similar chunks238    ↓239Send to Groq API with context240    ↓241Groq generates answer242    ↓243Return to frontend244    ↓245Display in chat246```247 248---249 250## Development Workflow251 252### Local Development253 254```bash255# Setup256python -m venv venv257source venv/bin/activate258pip install -r requirements.txt259 260# Run261export GROQ_API_KEY=your_key262python app.py263 264# Access265http://localhost:7860266 267# Debug268tail -f app.log269# or set FLASK_ENV=development for auto-reload270```271 272### Docker Development273 274```bash275# Build276docker build -t graph-rag .277 278# Run with logs279docker run -p 7860:7860 \280  -e GROQ_API_KEY=your_key \281  -v $(pwd)/data:/app/data \282  graph-rag283 284# Or use Compose285docker-compose up --build286```287 288### Testing289 290```bash291# See TESTING.md for detailed test cases292# Quick test: manual UI testing293# Run: navigate to http://localhost:7860294# Steps: upload → visualize → query295```296 297---298 299## Customization Points300 301### Easy Customizations302 3031. **Styling**: Edit `templates/index.html` CSS section (lines 15-300)3042. **Colors**: Change `#667eea` to your brand color (all occurrences)3053. **Title**: Change "Graph RAG Chatbot" in HTML title and headers3064. **Icons**: Replace emoji with SVG icons3075. **Fonts**: Add Google Fonts in `<head>`308 309### Moderate Customizations310 3111. **Chunk Size**: `app.py` line 663122. **Embedding Model**: `app.py` line 273133. **LLM Model**: `app.py` line 1533144. **Similarity Threshold**: `app.py` line 1643155. **Graph Layout**: `app.py` NetworkX spring_layout parameters316 317### Advanced Customizations318 3191. **Database**: Replace in-memory `documents_state` with PostgreSQL3202. **Vector Storage**: Add ChromaDB or Pinecone3213. **Authentication**: Add user login with Flask-Login3224. **Caching**: Add Redis for embedding cache3235. **Monitoring**: Add Prometheus metrics324 325---326 327## Deployment Targets328 329| Target | Path | Docs |330|---|---|---|331| Local | Direct Python | README.md |332| Local Docker | Docker | README.md |333| HF Spaces | Auto-deploy | space_config.md |334| AWS | ECR → ECS | README.md |335| Azure | ACR → App Service | README.md |336| GCP | Artifact Registry | README.md |337| DigitalOcean | App Platform | README.md |338 339---340 341## Performance Characteristics342 343### Startup344- Cold start: 30-60s (model download)345- Warm start: 2-3s (in-memory)346- Model size: ~400MB347 348### Upload Processing349- Small file (< 5MB): 5-10s350- Medium file (5-20MB): 15-30s351- Large file (20-50MB): 30-60s352 353### Query Response354- Embedding: 0.5-1s355- Similarity search: <0.1s356- LLM generation: 1-3s357- Total: 2-5s358 359### Concurrency360- Single-threaded requests: No361- Async upload: Yes (threading)362- Parallel documents: Yes (3+ simultaneous)363 364---365 366## Security Considerations367 368### API Security369- ✅ No API authentication (add if needed)370- ✅ CORS enabled (all origins)371- ✅ File size limit: 50MB372- ✅ Groq API key not exposed to frontend373 374### Data Security375- ✅ Files stored server-side only376- ✅ No sensitive data logging377- ✅ Uploaded files deleted on request378- ⚠️  No encryption at rest (add for sensitive data)379 380### Deployment Security381- ✅ Python 3.11-slim base (minimal OS)382- ✅ No root user in container383- ✅ .env not committed384- ✅ Health checks enabled385 386---387 388## Known Limitations389 3901. **Storage**: Ephemeral (HF Spaces free tier)391   - Solution: Upgrade to persistent storage392 3932. **Processing Speed**: Single machine394   - Solution: Use GPU tier or distributed processing395 3963. **Concurrency**: Threading (Python GIL)397   - Solution: Use Gunicorn with multiple workers398 3994. **Graph Complexity**: Limited to 500 nodes400   - Solution: Implement hierarchical graph layouts401 4025. **API Rate Limits**: Groq free tier 30req/min403   - Solution: Implement caching or upgrade plan404 405---406 407## Future Enhancements408 409- [ ] WebSocket for real-time updates410- [ ] Database backend (PostgreSQL + pgvector)411- [ ] Multi-user with authentication412- [ ] Advanced graph algorithms (pagerank, centrality)413- [ ] Export to PDF/HTML reports414- [ ] Multi-language support415- [ ] Fine-tuned embeddings model416- [ ] Conversation memory/history417- [ ] Advanced search (filters, facets)418- [ ] API documentation (Swagger/OpenAPI)419 420---421 422## File Ownership & Maintenance423 424| File | Created | Last Updated | Maintainer |425|---|---|---|---|426| app.py | Day 1 | Day 1 | You |427| index.html | Day 1 | Day 1 | You |428| Dockerfile | Day 1 | Day 1 | You |429| requirements.txt | Day 1 | Day 1 | You |430| README.md | Day 1 | Day 1 | You |431 432---433 434**Total Project Size**: ~5MB (including dependencies on first run: ~2GB)435**Source Code Size**: ~50KB (uncompressed)436**Docker Image Size**: ~2.5GB (uncompressed)437**Docker Image Size**: ~800MB (compressed)438 439---440 441**Last Updated**: June 27, 2024442**Version**: 1.0.0443**Status**: Production Ready ✅444