findEthics/Atlas

1# Search Optimizer Developer Guide2 3## Overview4 5The Atlas search optimizer is a sophisticated system that intelligently determines when web searches are necessary based on conversation context, user intent, and available information. This guide covers the internal architecture, functions, and customization options for developers.6 7## Architecture8 9### Core Components10 11The search optimization system consists of several interconnected components:12 13```14┌─────────────────────────────────────────────────────────────┐15│                    Chat Endpoint                            │16└─────────────────────┬───────────────────────────────────────┘17                      │18┌─────────────────────▼───────────────────────────────────────┐19│               Request Flow Router                           │20│  ┌─────────────────┐    ┌─────────────────────────────────┐ │21│  │  Cache-First    │    │  Search-Decision-First          │ │22│  │  (No History)   │    │  (Has History)                  │ │23│  └─────────────────┘    └─────────────────────────────────┘ │24└─────────────────────┬───────────────────┬───────────────────┘25                      │                   │26┌─────────────────────▼───────────────────▼───────────────────┐27│                Hybrid Search Engine                         │28│  ┌───────────────┐  ┌──────────────┐  ┌──────────────────┐ │29│  │ Rule-Based    │  │ AI Analysis  │  │ Context Analysis │ │30│  │ Patterns      │  │ (Gemini)     │  │ (spaCy)          │ │31│  └───────────────┘  └──────────────┘  └──────────────────┘ │32└─────────────────────┬───────────────────────────────────────┘33                      │34┌─────────────────────▼───────────────────────────────────────┐35│                ChromaDB Cache                               │36│            Semantic Vector Matching                        │37└─────────────────────────────────────────────────────────────┘38```39 40### Module Structure41 42```python43search_optimizer.py44├── SearchOptimizer class          # Main interface45├── Rule-based functions           # Pattern matching46│   ├── should_perform_search()47│   ├── has_meaningful_conversation_history()48│   └── analyze_conversation_context()49├── AI-based functions            # Intelligent analysis  50│   └── analyze_search_necessity()51├── Hybrid engine                 # Combined decision making52│   └── hybrid_search_decision()53└── Utility functions             # Support functions54    ├── extract_search_terms()55    ├── format_search_context()56    └── clean_terms()57```58 59## Core Functions60 61### Rule-Based Search Decision62 63#### `should_perform_search(prompt, history, search_decision_mode)`64 65**Purpose**: Fast pattern-based search decision using linguistic rules.66 67**Parameters**:68- `prompt: str` - User's current question69- `history: Optional[List[Dict[str, str]]]` - Conversation history  70- `search_decision_mode: str` - Sensitivity mode ("conservative", "balanced", "aggressive")71 72**Returns**: `Dict[str, Any]` with:73- `should_search: bool` - Search decision74- `reason: str` - Explanation75- `confidence: float` - Decision confidence (0.0-1.0)76 77**Pattern Detection**:78```python79# Elaboration patterns80["elaborate", "explain more", "tell me more", "expand on", "go deeper"]81 82# Clarification patterns  83["what do you mean", "can you clarify", "i don't understand", "unclear"]84 85# Referential patterns86["this", "that", "it", "the previous", "above mentioned", "earlier"]87 88# Continuation patterns89["and what about", "what else", "continue", "also", "additionally"]90```91 92**Example Usage**:93```python94from search_optimizer import should_perform_search95 96result = should_perform_search(97    "Tell me more about neural networks",98    [{"user": "What is AI?", "assistant": "AI is artificial intelligence..."}],99    "balanced"100)101 102# Result: {"should_search": False, "reason": "Elaboration request with existing context", "confidence": 0.7}103```104 105### AI-Based Search Analysis106 107#### `analyze_search_necessity(prompt, history, conversation_context, gemini_model)`108 109**Purpose**: Deep AI-powered analysis for ambiguous cases.110 111**Features**:112- Question type classification (new info vs clarification)113- Information sufficiency assessment114- Topic continuity detection  115- Recency requirements analysis116- Semantic similarity scoring117 118**Caching**: Results cached for 5 minutes to optimize performance.119 120**Example Usage**:121```python122from search_optimizer import analyze_search_necessity123 124result = await analyze_search_necessity(125    "What are the latest developments?",126    history,127    conversation_context,128    gemini_model129)130 131# Result includes detailed analysis breakdown132```133 134### Hybrid Decision Engine135 136#### `hybrid_search_decision(prompt, history, search_decision_mode, nlp_model, gemini_model)`137 138**Purpose**: Combines rule-based and AI analysis for optimal decisions.139 140**Decision Logic**:1411. **Rule-based first**: Fast pattern matching1422. **Confidence check**: Use AI if rule confidence < threshold1433. **Weighted combination**: Merge rule and AI decisions1444. **Context analysis**: Add semantic similarity metrics145 146**Thresholds by Mode**:147- `conservative`: AI threshold 0.8 (prefer rules)  148- `balanced`: AI threshold 0.6 (balanced approach)149- `aggressive`: AI threshold 0.4 (prefer AI analysis)150 151### Context Analysis152 153#### `analyze_conversation_context(prompt, history, nlp_model)`154 155**Purpose**: Semantic analysis of conversation continuity.156 157**Metrics Calculated**:158- `topic_continuity`: Keyword overlap score159- `semantic_similarity`: spaCy vector similarity  160- `information_coverage`: History richness score161- `context_richness`: Overall context quality162 163**spaCy Integration**: Uses `en_core_web_sm` model for:164- Word vectors and similarity165- Lemmatization and tokenization166- Stop word filtering167- Dependency parsing168 169### Utility Functions170 171#### `extract_search_terms(text, nlp_model, rake_instance)`172 173**NLP Pipeline**:1741. **Named Entity Recognition**: Extract proper nouns1752. **Noun Phrase Extraction**: Syntactic analysis  1763. **RAKE Keywords**: Top-ranked phrases1774. **Focus Phrase Detection**: Dependency parsing1785. **Term Cleaning**: Deduplication and filtering179 180**Example**:181```python182terms = extract_search_terms(183    "What is machine learning in healthcare?",184    nlp, rake185)186# Returns: ["machine learning", "healthcare", "machine learning healthcare"]187```188 189#### `format_search_context(results)`190 191**Purpose**: Format search results for AI consumption.192 193**Features**:194- Source attribution  195- Content truncation (1200 chars per result)196- Error handling for malformed results197- Structured output for AI processing198 199## Configuration & Customization200 201### Search Decision Modes202 203**Conservative Mode**:204- Elaboration threshold: 0.8 (high confidence required)205- Referential threshold: 0.7206- History weight: 0.9 (heavily favor existing context)207 208**Balanced Mode**:209- Elaboration threshold: 0.6  210- Referential threshold: 0.5211- History weight: 0.7212 213**Aggressive Mode**:214- Elaboration threshold: 0.4 (low confidence required)215- Referential threshold: 0.3  216- History weight: 0.5 (prefer fresh searches)217 218### Pattern Customization219 220Add custom patterns to the decision logic:221 222```python223# In should_perform_search function224custom_patterns = [225    "help me understand",226    "break down",227    "simplify this"228]229 230elaboration_patterns.extend(custom_patterns)231```232 233### AI Prompt Customization234 235Modify the AI analysis prompt in `analyze_search_necessity`:236 237```python238analysis_prompt = f"""239Analyze whether a web search is necessary for: {prompt}240 241Custom criteria:2421. Domain-specific requirements2432. Company knowledge base availability  2443. User expertise level245 246Respond with JSON: {{"should_search": bool, "confidence": float, "reason": str}}247"""248```249 250## Performance Optimization251 252### Caching Strategy253 254**AI Decision Cache**:255- 5-minute TTL for search decisions256- Hash-based keys using prompt + history257- Automatic cleanup and memory management258 259**ChromaDB Vector Cache**:260- Persistent storage across restarts261- Semantic similarity matching (threshold 0.7)262- TTL-based expiration with cleanup263 264### Performance Monitoring265 266Track key metrics:267 268```python269# Function execution times270hybrid_decision_time = measure_time(hybrid_search_decision)271 272# Cache hit rates273cache_stats = search_cache.get_stats()274hit_rate = cache_stats["hit_rate_percentage"]275 276# AI analysis frequency  277ai_calls_percentage = ai_decisions / total_decisions278```279 280### Optimization Guidelines281 2821. **Rule-based first**: Fast patterns handle 60-70% of cases2832. **Cache aggressively**: ChromaDB for search results, memory for decisions2843. **Monitor thresholds**: Adjust AI confidence thresholds based on usage2854. **Batch operations**: Group similar requests when possible286 287## Integration Patterns288 289### Basic Integration290 291```python292from search_optimizer import hybrid_search_decision293 294# In your chat endpoint295search_decision = await hybrid_search_decision(296    request.prompt,297    request.history, 298    request.search_decision_mode,299    nlp_model,300    gemini_model301)302 303if search_decision["should_search"]:304    # Perform web search305    search_results = await search_web_combined(query)306else:307    # Use conversation history only308    search_results = []309```310 311### Advanced Integration312 313```python314# Custom decision logic315async def custom_search_decision(request, models):316    # Step 1: Check force_search override317    if request.force_search is not None:318        return {"should_search": request.force_search, "reason": "User override"}319    320    # Step 2: Domain-specific rules321    if is_internal_knowledge(request.prompt):322        return {"should_search": False, "reason": "Internal knowledge available"}323    324    # Step 3: Use hybrid engine325    return await hybrid_search_decision(326        request.prompt, request.history,327        request.search_decision_mode, 328        models.nlp, models.gemini329    )330```331 332### Error Handling333 334```python335try:336    search_decision = await hybrid_search_decision(...)337except Exception as e:338    logger.error(f"Search decision failed: {e}")339    # Fallback to safe default340    search_decision = {341        "should_search": True,342        "reason": f"Decision engine error: {str(e)[:50]}",343        "confidence": 0.5,344        "decision_method": "fallback"345    }346```347 348## Testing & Validation349 350### Unit Testing351 352```python353def test_search_decision_patterns():354    # Test elaboration detection355    result = should_perform_search("Tell me more", history)356    assert not result["should_search"]357    assert "elaboration" in result["reason"].lower()358    359    # Test new information requests360    result = should_perform_search("What's the latest news?", None)361    assert result["should_search"]362    assert result["confidence"] > 0.8363```364 365### Integration Testing366 367```python368async def test_hybrid_engine():369    # Test rule-based path370    result = await hybrid_search_decision("Hello", [], "balanced", nlp, model)371    assert result["decision_method"] == "rule_based"372    373    # Test hybrid path  374    result = await hybrid_search_decision(ambiguous_prompt, [], "balanced", nlp, model)375    assert result["decision_method"] == "hybrid"376```377 378### Performance Testing379 380```python381def benchmark_search_decisions():382    import time383    384    start = time.time()385    for _ in range(100):386        should_perform_search("test prompt", [])387    end = time.time()388    389    avg_time = (end - start) / 100390    assert avg_time < 0.01  # Sub-10ms performance391```392 393## Monitoring & Debugging394 395### Logging Integration396 397```python398import logging399logger = logging.getLogger("search_optimizer")400 401# Enable debug logging402logger.setLevel(logging.DEBUG)403 404# In functions, use structured logging405logger.info(f"Search decision: {decision['should_search']}, confidence: {decision['confidence']:.2f}, reason: {decision['reason']}")406```407 408### Metrics Collection409 410```python411# Decision distribution412rule_based_count = 0413hybrid_count = 0  414ai_only_count = 0415 416# Performance metrics417decision_times = []418cache_hit_rates = []419false_positive_rate = 0.0  # Search when not needed420false_negative_rate = 0.0  # No search when needed421```422 423### Debug Utilities424 425```python426def debug_search_decision(prompt, history):427    """Detailed debugging for search decisions"""428    429    print(f"Analyzing: {prompt}")430    print(f"History entries: {len(history or [])}")431    432    # Rule-based analysis433    rule_result = should_perform_search(prompt, history)434    print(f"Rule decision: {rule_result}")435    436    # Context analysis437    context = analyze_conversation_context(prompt, history, nlp)438    print(f"Context metrics: {context}")439    440    # Final decision441    final_result = await hybrid_search_decision(prompt, history, "balanced", nlp, model)442    print(f"Final decision: {final_result}")443```444 445## Future Enhancements446 447### Planned Features448 4491. **Machine Learning Integration**: Train models on decision patterns4502. **User Behavior Analysis**: Personalized search thresholds  4513. **Domain-Specific Rules**: Industry/topic-specific optimization4524. **Multimodal Support**: Image and document context analysis4535. **Real-time Learning**: Adaptive thresholds based on feedback454 455### Extension Points456 457```python458# Custom analyzers459class CustomSearchAnalyzer:460    def analyze(self, prompt, history, context):461        # Custom analysis logic462        return {"should_search": bool, "confidence": float}463 464# Plugin architecture465search_plugins = [466    DomainSpecificAnalyzer(),467    UserBehaviorAnalyzer(), 468    CustomSearchAnalyzer()469]470```471 472## Conclusion473 474The search optimizer provides a robust, intelligent system for minimizing unnecessary web searches while maintaining response quality. The hybrid approach combining rule-based patterns with AI analysis offers both performance and accuracy.475 476Key benefits:477- **40-60% reduction** in unnecessary searches478- **Sub-millisecond** rule-based decisions479- **Intelligent fallbacks** for edge cases480- **Comprehensive caching** for performance481- **Extensive customization** options482 483For questions or contributions, refer to the main Atlas documentation or create issues in the project repository.