findEthics/Atlas

1# Atlas Setup Guide2 3## Overview4Atlas is an enhanced chat API service that provides intelligent question-answering capabilities with web search augmentation and comprehensive analytics. It uses Google's Gemini model, combines multiple search engines for comprehensive results, and includes a full analytics dashboard with MongoDB integration for session and message tracking.5 6## Virtual Environment Setup7 8The project has been set up with a Python virtual environment using the specifications from the Dockerfile:9 10- **Python Version**: 3.13.5 (newer than the 3.9 specified in Dockerfile)11- **Virtual Environment**: `atlas_env`12- **All dependencies**: Successfully installed13 14## Environment Variables15 16Create a `.env` file in the project root with the following variables:17 18```bash19# Required: Google API Key for Gemini model20GOOGLE_API_KEY=your_google_api_key_here21 22# Optional: Brave Search API Key (falls back to DuckDuckGo if not provided)23BRAVE_API_KEY=your_brave_api_key_here24 25# Required: MongoDB Configuration for Analytics26MONGODB_URL=mongodb+srv://username:password@cluster.mongodb.net/?retryWrites=true&w=majority&appName=Atlas27MONGODB_DATABASE=Atlas28 29# Application Settings (optional - defaults are used if not set)30PORT=786031HOST=0.0.0.032```33 34### Getting API Keys and Database Setup35 361. **Google API Key**: 37   - Go to [Google AI Studio](https://makersuite.google.com/app/apikey)38   - Create a new API key39   - Add it to your `.env` file40 412. **Brave Search API Key** (Optional):42   - Go to [Brave Search API](https://api.search.brave.com/)43   - Sign up and get your API key44   - Add it to your `.env` file45 463. **MongoDB Atlas Setup** (Required for Analytics):47   - Go to [MongoDB Atlas](https://www.mongodb.com/atlas)48   - Create a free account and cluster49   - Create a database user with read/write permissions50   - Get your connection string and add it to your `.env` file51   - The analytics system requires MongoDB for session and message tracking52 53## Running the Application54 55### Option 1: Using the startup script56```bash57./start.sh58```59 60### Option 2: Manual startup61```bash62# Activate virtual environment63source atlas_env/bin/activate64 65# Run the application66python app.py67```68 69### Option 3: Using uvicorn directly70```bash71# Activate virtual environment72source atlas_env/bin/activate73 74# Run with uvicorn75uvicorn app:app --host 0.0.0.0 --port 786076```77 78## Testing the Setup79 80### MongoDB Connection Test81Verify your MongoDB connection is working:82 83```bash84python test_mongo_connection.py85```86 87This will test:88- MongoDB connection with both sync and async drivers89- Database accessibility90- Environment variable configuration91 92### Quick Setup Verification93You can also verify the setup by starting the server and checking the health endpoint:94 95```bash96# Start the server97./start.sh98 99# In another terminal, test the health endpoint100curl http://localhost:7860/101```102 103## API Endpoints104 105Once running, the application provides these endpoints:106 107### Core Functionality108- **`/`** - Health check and status109- **`/chat`** - Main chat endpoint with search augmentation110- **`/search`** - Direct search functionality111- **`/docs`** - Interactive API documentation (Swagger UI)112 113### Analytics & Cache Management114- **`/analytics/stats`** - JSON API with analytics statistics115- **`/analytics/dashboard`** - Interactive HTML dashboard with charts116- **`/analytics/export`** - Export analytics data (JSON/CSV format)117- **`/analytics/cache`** - Cache performance metrics and statistics118- **`/analytics/cache/clear`** - Cache management and maintenance119- **`/analytics/users`** - User statistics and anonymous vs authenticated metrics120- **`/analytics/user/{user_id}`** - Individual user analytics and insights121- **`/analytics/comparison`** - Detailed authenticated vs anonymous comparison122 123### Example Usage124 125#### Anonymous Mode (No Authentication Required)126```bash127# Health check128curl http://localhost:7860/129 130# Simple anonymous chat request131curl -X POST http://localhost:7860/chat \132  -H "Content-Type: application/json" \133  -d '{"prompt": "What is artificial intelligence?", "use_search": true}'134 135# Anonymous request without search136curl -X POST http://localhost:7860/chat \137  -H "Content-Type: application/json" \138  -d '{"prompt": "What is 2+2?", "use_search": false}'139 140# Anonymous request with search optimization control141curl -X POST http://localhost:7860/chat \142  -H "Content-Type: application/json" \143  -d '{144    "prompt": "What are the latest AI developments?",145    "search_decision_mode": "aggressive",146    "force_search": true147  }'148 149# Anonymous request with conversation history150curl -X POST http://localhost:7860/chat \151  -H "Content-Type: application/json" \152  -d '{153    "prompt": "Can you elaborate on that?",154    "use_search": false,155    "history": [156      {"role": "user", "content": "What is machine learning?"},157      {"role": "assistant", "content": "Machine learning is a subset of AI..."}158    ]159  }'160```161 162#### Authenticated Mode (With User Tracking)163```bash164# Authenticated chat request165curl -X POST http://localhost:7860/chat \166  -H "Content-Type: application/json" \167  -d '{168    "prompt": "What is my chat history?",169    "user_id": "test-user-123",170    "use_search": true171  }'172 173# Authenticated request with session continuity174curl -X POST http://localhost:7860/chat \175  -H "Content-Type: application/json" \176  -H "X-Session-ID: session-uuid-here" \177  -d '{178    "prompt": "Continue our previous conversation",179    "user_id": "test-user-123",180    "use_search": false181  }'182```183 184#### Analytics & Monitoring185```bash186# View analytics (includes anonymous vs authenticated breakdown)187curl http://localhost:7860/analytics/stats188 189# Export analytics data190curl "http://localhost:7860/analytics/export?format=json&days=7"191 192# View cache performance metrics193curl http://localhost:7860/analytics/cache194 195# Clear expired cache entries196curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired197 198# View user statistics breakdown199curl http://localhost:7860/analytics/users200 201# View specific user analytics  202curl http://localhost:7860/analytics/user/user123203 204# Access interactive dashboard in browser205open http://localhost:7860/analytics/dashboard206```207 208## Features209 210### 🤖 AI-Powered Chat211- Uses Google's Gemini 1.5 Flash model212- Configurable parameters (temperature, max tokens)213- Intelligent responses based on web search results214- Session-based conversation tracking215 216### 🔍 Advanced Web Search & Optimization217- **Dual Search Engine Strategy**: Brave Search + DuckDuckGo218- **Resilient Fallback**: Automatic fallback if one engine fails219- **Smart Query Extraction**: NLP-powered search term extraction using spaCy and RAKE220- **Deduplication**: Removes duplicate results across engines221- **🧠 Intelligent Search Optimization**: AI-powered search decision engine222- **⚡ Context-Aware Flow**: Cache-first for new conversations, smart decisions for follow-ups223- **🗄️ ChromaDB Vector Caching**: Semantic similarity matching with persistent storage224- **📊 Search Analytics**: Comprehensive search decision and performance tracking225 226### 🧠 NLP-Powered Processing227- Named Entity Recognition228- Dependency parsing for question focus229- RAKE keyword extraction230- Text preprocessing and lemmatization231 232### 📊 Comprehensive Analytics233- **Real-time Session Tracking**: Monitor user sessions and activity234- **Message Analytics**: Track response times, search usage, and success rates235- **Interactive Dashboard**: Beautiful HTML dashboard with charts and metrics236- **Data Export**: Export analytics data in JSON or CSV format237- **MongoDB Integration**: Persistent storage for all analytics data238- **Performance Monitoring**: Response time percentiles and error tracking239 240## Troubleshooting241 242### Common Issues243 2441. **Import Errors**: Make sure you're in the virtual environment245   ```bash246   source atlas_env/bin/activate247   ```248 2492. **API Key Errors**: Check your `.env` file and ensure API keys are set correctly250 2513. **MongoDB Connection Issues**: 252   - Verify your `MONGODB_URL` is correct in `.env`253   - Check your MongoDB Atlas cluster is running254   - Ensure your IP address is whitelisted in MongoDB Atlas255   - Test connection with: `python test_mongo_connection.py`256 2574. **spaCy Model Issues**: The model should be automatically downloaded, but you can manually download it:258   ```bash259   python -m spacy download en_core_web_sm260   ```261 2625. **NLTK Data Issues**: NLTK data is automatically downloaded on first run263 2646. **Analytics Not Working**: 265   - Check MongoDB connection266   - Verify environment variables are loaded267   - Restart the server after updating `.env`268 269### Port Conflicts270 271If port 7860 is already in use, you can change it in the `.env` file or run with a different port:272 273```bash274uvicorn app:app --host 0.0.0.0 --port 8000275```276 277## Development278 279### Adding New Dependencies280 2811. Add to `requirements.txt`2822. Install in virtual environment:283   ```bash284   source atlas_env/bin/activate285   pip install -r requirements.txt286   ```287 288### Testing Changes289 290- Use `python test_mongo_connection.py` to verify MongoDB connectivity291- Check the health endpoint at `http://localhost:7860/` after starting the server292- Monitor the analytics dashboard at `http://localhost:7860/analytics/dashboard`293 294### Current Dependencies295 296The project includes these key packages:297- `fastapi` - Web framework298- `motor` - Async MongoDB driver299- `google-generativeai` - Google Gemini API300- `spacy` - NLP processing301- `nltk` - Natural language toolkit302- `duckduckgo-search` - Web search303- `httpx` - HTTP client for Brave Search304 305## Production Deployment306 307For production deployment, consider:308- Using the provided Dockerfile309- Setting up proper environment variables (especially secure MongoDB credentials)310- Configuring reverse proxy (nginx)311- Setting up monitoring and logging312- Using a process manager (systemd, supervisor)313- Implementing proper MongoDB security (authentication, network restrictions)314- Setting up MongoDB backups for analytics data315- Configuring CORS properly for your domain316 317## Support318 319If you encounter issues:3201. Run `python test_mongo_connection.py` to test MongoDB connectivity3212. Check the server logs for error messages3223. Verify all environment variables are set correctly in `.env`3234. Ensure you're using the virtual environment3245. Test the health endpoint: `curl http://localhost:7860/`3256. Check the analytics dashboard for system status3267. Verify your MongoDB Atlas cluster is running and accessible327 328## Analytics System329 330The analytics system provides comprehensive insights into your chat application usage:331 332### Features333- **Session Tracking**: Each user interaction creates a session with unique ID334- **Message Analytics**: Response times, search usage, success rates335- **Real-time Dashboard**: Interactive charts and statistics336- **Data Export**: Download analytics data for external analysis337- **Performance Monitoring**: Track system performance and errors338 339### Accessing Analytics340- **Dashboard**: `http://localhost:7860/analytics/dashboard`341- **API**: `http://localhost:7860/analytics/stats`342- **Export**: `http://localhost:7860/analytics/export?format=json&days=7`343 344### Data Collected345- Session information (start time, duration, message count)346- Message metrics (prompt length, response time, search usage)347- Performance data (response time percentiles, error rates)348- Search analytics (engine usage, result counts)