findEthics/Atlas

1# Search Optimization Troubleshooting Guide2 3## Overview4 5This guide helps diagnose and resolve issues with Atlas's search optimization features, including smart search decisions, caching problems, and performance issues.6 7## Common Issues8 9### Search Decision Problems10 11#### Issue: Too Many Unnecessary Searches12 13**Symptoms**:14- High search API costs15- Slow response times for follow-up questions16- `search_decision.should_search` is true for obvious follow-ups17 18**Diagnosis**:19```bash20# Check recent search decisions21curl http://localhost:7860/analytics/stats | jq '.search_usage_percentage'22 23# View specific decision details in responses24curl -X POST http://localhost:7860/chat -d '{25  "prompt": "Tell me more about that",26  "history": [{"role": "user", "content": "What is AI?"}]27}' | jq '.search_decision'28```29 30**Solutions**:31 321. **Use Conservative Mode**:33```json34{35  "prompt": "Follow-up question",36  "search_decision_mode": "conservative"37}38```39 402. **Check Conversation History Format**:41```javascript42// Correct format43history: [44  {"role": "user", "content": "What is AI?"},45  {"role": "assistant", "content": "AI is artificial intelligence..."}46]47 48// Alternative format  49history: [50  {"user": "What is AI?", "assistant": "AI is artificial intelligence..."}51]52 53// Incorrect - will cause unnecessary searches54history: [55  {"message": "What is AI?", "response": "AI is..."}  // Wrong keys56]57```58 593. **Verify NLP Dependencies**:60```bash61# Check if spaCy model is loaded62python -c "import spacy; nlp = spacy.load('en_core_web_sm'); print('spaCy OK')"63 64# Check RAKE installation65python -c "from rake_nltk import Rake; print('RAKE OK')"66```67 68#### Issue: Missing Important Information69 70**Symptoms**:71- Outdated responses for current events72- `search_decision.should_search` is false for time-sensitive queries73- Users complaining about stale information74 75**Diagnosis**:76```bash77# Check search decision patterns78curl -X POST http://localhost:7860/chat -d '{79  "prompt": "What are today'\''s tech news?",80  "search_decision_mode": "balanced"81}' | jq '.search_decision'82```83 84**Solutions**:85 861. **Use Aggressive Mode for Current Events**:87```json88{89  "prompt": "Latest developments in AI",90  "search_decision_mode": "aggressive"91}92```93 942. **Force Search for Critical Updates**:95```json96{97  "prompt": "Current stock price of AAPL", 98  "force_search": true99}100```101 1023. **Add Recency Keywords**:103```json104{105  "prompt": "What are the latest news about climate change today?"106}107// Keywords like "latest", "today", "current" trigger searches108```109 110#### Issue: Inconsistent Search Decisions111 112**Symptoms**:113- Similar questions get different search decisions114- `search_decision.confidence` is very low (< 0.5)115- Decision method frequently falls back to "fallback_rule"116 117**Diagnosis**:118```bash119# Test decision consistency120for i in {1..5}; do121  curl -X POST http://localhost:7860/chat -d '{122    "prompt": "Tell me more about machine learning"123  }' | jq '.search_decision.should_search'124done125```126 127**Solutions**:128 1291. **Check AI Model Availability**:130```python131# Verify Gemini API key132import os133print("GOOGLE_API_KEY:", "✓" if os.getenv("GOOGLE_API_KEY") else "✗")134```135 1362. **Monitor AI Decision Cache**:137```bash138# Clear AI decision cache if stale139curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=all140```141 1423. **Review Conversation History Quality**:143```javascript144// Ensure meaningful history entries145const validHistory = history.filter(entry => 146  entry.role && entry.content && entry.content.length > 10147);148```149 150### Cache-Related Issues151 152#### Issue: Poor Cache Hit Rates153 154**Symptoms**:155- `cache_info.cache_hit` is frequently false156- High response times despite caching157- Cache hit rate < 30% in analytics158 159**Diagnosis**:160```bash161# Check cache performance162curl http://localhost:7860/analytics/cache | jq '{163  hit_rate: .cache_statistics.hit_rate_percentage,164  cache_size: .cache_statistics.cache_size,165  effectiveness: .cache_effectiveness166}'167```168 169**Solutions**:170 1711. **Check ChromaDB Configuration**:172```bash173# Verify ChromaDB dependencies174python -c "import chromadb; print('ChromaDB OK')"175python -c "from sentence_transformers import SentenceTransformer; print('SentenceTransformers OK')"176```177 1782. **Adjust Similarity Threshold**:179```python180# In cache configuration (environment variables)181CACHE_SIMILARITY_THRESHOLD=0.6  # Lower = more hits, less precision182CACHE_SIMILARITY_THRESHOLD=0.8  # Higher = fewer hits, more precision183```184 1853. **Monitor Query Patterns**:186```bash187# View popular queries188curl http://localhost:7860/analytics/cache | jq '.popular_queries'189```190 191#### Issue: Cache Storage Problems192 193**Symptoms**:194- `cache_info.stored_in_cache` is false195- Cache size not growing196- Persistent storage not working across restarts197 198**Diagnosis**:199```bash200# Check cache directory permissions201ls -la cache_db/202ls -la cache_results/203 204# Check disk space205df -h .206```207 208**Solutions**:209 2101. **Fix Directory Permissions**:211```bash212mkdir -p cache_db cache_results213chmod 755 cache_db cache_results214```215 2162. **Check Environment Variables**:217```bash218# Verify cache configuration219echo $CHROMADB_PATH220echo $CACHE_RESULTS_PATH221echo $CACHE_EMBEDDING_MODEL222```223 2243. **Clear Corrupted Cache**:225```bash226# Stop server, clear cache, restart227rm -rf cache_db cache_results228curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=all229```230 231#### Issue: Memory Usage Issues232 233**Symptoms**:  234- High memory consumption235- `cache_statistics.memory_usage_mb` increasing rapidly236- Server running out of memory237 238**Diagnosis**:239```bash240# Monitor cache memory usage241curl http://localhost:7860/analytics/cache | jq '.cache_statistics.memory_usage_mb'242 243# Check system memory244free -h245ps aux | grep python246```247 248**Solutions**:249 2501. **Adjust Cache Size Limits**:251```python252# In cache configuration253max_cache_size = 500  # Reduce from default 1000254```255 2562. **Implement Regular Cleanup**:257```bash258# Schedule cache cleanup259curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired260```261 2623. **Monitor Cache Efficiency**:263```bash264# Check entries per MB ratio265curl http://localhost:7860/analytics/cache | jq '.memory_efficiency'266```267 268### Performance Issues269 270#### Issue: Slow Search Decisions271 272**Symptoms**:273- Response times > 5 seconds for simple questions274- `decision_method` frequently shows "hybrid" or AI usage275- High CPU usage during decision making276 277**Diagnosis**:278```bash279# Time search decision performance280time curl -X POST http://localhost:7860/chat -d '{281  "prompt": "Simple question"282}' > /dev/null283```284 285**Solutions**:286 2871. **Optimize for Rule-Based Decisions**:288```json289{290  "search_decision_mode": "balanced"  // Uses more rules, less AI291}292```293 2942. **Check NLP Model Performance**:295```python296import time297import spacy298 299nlp = spacy.load("en_core_web_sm")300start = time.time()301doc = nlp("test sentence")302print(f"spaCy processing time: {time.time() - start:.3f}s")303```304 3053. **Monitor AI API Latency**:306```python307# Check Gemini API response times308import time309import google.generativeai as genai310 311start = time.time()  312response = model.generate_content("test")313print(f"Gemini latency: {time.time() - start:.3f}s")314```315 316#### Issue: High Memory Usage317 318**Symptoms**:319- Gradual memory increase over time320- Server crashes with out-of-memory errors321- Slow performance after extended usage322 323**Diagnosis**:324```bash325# Monitor memory usage patterns326ps aux | grep -E 'python|atlas' | head -5327 328# Check for memory leaks329curl http://localhost:7860/analytics/cache | jq '.cache_statistics'330```331 332**Solutions**:333 3341. **Implement Cache Limits**:335```python336# Set maximum cache entries337MAX_CACHE_ENTRIES = 1000338MAX_MEMORY_MB = 100339```340 3412. **Regular Cache Cleanup**:342```bash343# Automated cleanup script344#!/bin/bash345while true; do346  sleep 3600  # Every hour347  curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired348done349```350 3513. **Monitor Resource Usage**:352```bash353# Add monitoring script354watch 'curl -s http://localhost:7860/analytics/cache | jq ".cache_statistics.memory_usage_mb"'355```356 357## Diagnostic Tools358 359### Decision Analysis Script360 361```python362#!/usr/bin/env python3363"""Analyze search decision patterns"""364 365import requests366import json367 368def analyze_decisions(prompts):369    results = []370    for prompt in prompts:371        response = requests.post('http://localhost:7860/chat',372            json={'prompt': prompt}373        )374        data = response.json()375        results.append({376            'prompt': prompt,377            'should_search': data['search_decision']['should_search'],378            'reason': data['search_decision']['reason'],379            'confidence': data['search_decision']['confidence'],380            'method': data['search_decision'].get('decision_method')381        })382    return results383 384# Test cases385test_prompts = [386    "What is AI?",387    "Tell me more about that",388    "What's the latest news?",389    "Can you elaborate?",390    "How does machine learning work?"391]392 393results = analyze_decisions(test_prompts)394for result in results:395    print(f"'{result['prompt']}' -> {result['should_search']} ({result['confidence']:.2f}) - {result['reason']}")396```397 398### Cache Performance Monitor399 400```bash401#!/bin/bash402# Monitor cache performance over time403 404while true; do405  timestamp=$(date '+%Y-%m-%d %H:%M:%S')406  stats=$(curl -s http://localhost:7860/analytics/cache | jq '.cache_statistics')407  hit_rate=$(echo $stats | jq '.hit_rate_percentage')408  cache_size=$(echo $stats | jq '.cache_size')409  memory_mb=$(echo $stats | jq '.memory_usage_mb')410  411  echo "$timestamp - Hit Rate: ${hit_rate}%, Size: $cache_size, Memory: ${memory_mb}MB"412  sleep 60413done414```415 416### Health Check Script417 418```python419#!/usr/bin/env python3420"""Comprehensive health check for search optimization"""421 422import requests423import json424import sys425 426def health_check():427    issues = []428    429    # Check basic connectivity430    try:431        response = requests.get('http://localhost:7860/')432        if response.status_code != 200:433            issues.append("Server not responding correctly")434    except:435        issues.append("Cannot connect to server")436        return issues437    438    # Check search decision functionality439    try:440        response = requests.post('http://localhost:7860/chat', 441            json={'prompt': 'Test question'}442        )443        data = response.json()444        if 'search_decision' not in data:445            issues.append("Search decision not in response")446    except Exception as e:447        issues.append(f"Search decision error: {e}")448    449    # Check cache functionality  450    try:451        response = requests.get('http://localhost:7860/analytics/cache')452        if response.status_code != 200:453            issues.append("Cache analytics not working")454        else:455            cache_data = response.json()456            hit_rate = cache_data['cache_statistics']['hit_rate_percentage']457            if hit_rate < 10:458                issues.append(f"Very low cache hit rate: {hit_rate}%")459    except Exception as e:460        issues.append(f"Cache check error: {e}")461    462    # Check NLP dependencies463    try:464        import spacy465        nlp = spacy.load('en_core_web_sm')466    except Exception as e:467        issues.append(f"spaCy model error: {e}")468    469    try:470        from rake_nltk import Rake471    except Exception as e:472        issues.append(f"RAKE import error: {e}")473    474    return issues475 476if __name__ == "__main__":477    issues = health_check()478    if issues:479        print("❌ Issues found:")480        for issue in issues:481            print(f"  - {issue}")482        sys.exit(1)483    else:484        print("✅ All health checks passed")485        sys.exit(0)486```487 488## Configuration Troubleshooting489 490### Environment Variables491 492**Required Variables**:493```bash494# Essential for optimization495GOOGLE_API_KEY=your_key_here496 497# Cache configuration (optional)498CHROMADB_PATH=cache_db499CACHE_RESULTS_PATH=cache_results  500CACHE_EMBEDDING_MODEL=all-MiniLM-L6-v2501```502 503**Validation Script**:504```bash505#!/bin/bash506echo "Checking environment variables..."507 508if [ -z "$GOOGLE_API_KEY" ]; then509  echo "❌ GOOGLE_API_KEY not set"510else511  echo "✅ GOOGLE_API_KEY configured"512fi513 514if [ -d "$CHROMADB_PATH" ]; then515  echo "✅ ChromaDB path exists: $CHROMADB_PATH"516else517  echo "⚠️  ChromaDB path not found: $CHROMADB_PATH"518fi519```520 521### Dependency Issues522 523**Check All Dependencies**:524```python525#!/usr/bin/env python3526"""Check all optimization dependencies"""527 528dependencies = [529    ('spacy', 'spaCy NLP processing'),530    ('rake_nltk', 'RAKE keyword extraction'),531    ('chromadb', 'ChromaDB vector database'),532    ('sentence_transformers', 'Sentence embeddings'),533    ('google.generativeai', 'Google Gemini API')534]535 536for module, description in dependencies:537    try:538        __import__(module)539        print(f"✅ {description}: OK")540    except ImportError as e:541        print(f"❌ {description}: {e}")542```543 544## Performance Optimization545 546### Recommended Settings547 548**For High Traffic (Cost Optimization)**:549```json550{551  "search_decision_mode": "conservative",552  "cache_similarity_threshold": 0.6,553  "max_cache_size": 2000554}555```556 557**For Accuracy (Fresh Information)**:558```json559{560  "search_decision_mode": "aggressive", 561  "cache_similarity_threshold": 0.8,562  "force_search_for_news": true563}564```565 566**For Balanced Performance**:567```json568{569  "search_decision_mode": "balanced",570  "cache_similarity_threshold": 0.7,571  "ai_decision_timeout": 5.0572}573```574 575### Monitoring Metrics576 577**Key Metrics to Track**:578- Search reduction percentage (target: 40-60%)579- Cache hit rate (target: >50%)580- Response time improvement (target: 20-30% faster)581- Decision confidence (target: >0.7 average)582- False positive rate (searches when not needed: <5%)583- False negative rate (no search when needed: <5%)584 585**Monitoring Setup**:586```bash587# Create monitoring dashboard588curl http://localhost:7860/analytics/dashboard589 590# Set up alerting thresholds591if [ $(curl -s http://localhost:7860/analytics/cache | jq '.cache_statistics.hit_rate_percentage') < 30 ]; then592  echo "Alert: Low cache hit rate"593fi594```595 596## Getting Help597 598### Debug Information Collection599 600When reporting issues, include:601 6021. **System Information**:603```bash604python --version605pip list | grep -E 'spacy|chromadb|sentence|google'606df -h607free -h608```609 6102. **Configuration**:611```bash612env | grep -E 'GOOGLE|CACHE|CHROMADB'613ls -la cache_db/ cache_results/614```615 6163. **Recent Logs**:617```bash618# Server logs619tail -100 /var/log/atlas.log620 621# Decision patterns622curl http://localhost:7860/analytics/stats | jq '{623  search_usage: .search_usage_percentage,624  avg_response_time: .average_response_time_ms625}'626```627 6284. **Sample Requests**:629```bash630# Include problematic requests and responses631curl -X POST http://localhost:7860/chat -d '{632  "prompt": "Your problem prompt here"633}' | jq .634```635 636### Support Resources637 638- **Documentation**: [Search Optimization Guide](../features/search-optimization.md)639- **Developer Guide**: [Search Optimizer Developer Guide](../developer/search-optimizer-guide.md)640- **API Reference**: [Integration Guide](../api/integration-guide.md)641- **Performance**: [Setup Guide](../setup/SETUP.md)642 643For complex issues, create a detailed issue report with the debug information above and specific reproduction steps.