Aigenthix/Graph_RAG

1# RAG Comparison - Sample Benchmark Data2 3**Generated:** 2026-06-25  4**Models Tested:** Llama 3.1 8B, Llama 3.3 70B, GPT-OSS 120B, GPT-OSS 20B  5**Queries:** 50+ real-world scenarios6 7---8 9## 📊 Complete Dataset Structure10 11### Data Files Included in Package12 131. **benchmark_results.json** - Raw benchmark data142. **rag_comparison_metrics.csv** - Spreadsheet-friendly format153. **sample_documents/** - Test PDFs and CSVs164. **comparison_charts.json** - Pre-computed chart data17 18---19 20## 🔍 Sample Results by Query Type21 22### Query Category 1: Direct Fact Lookup23 24#### Query 1.1: "What is the company founded year?"25 26**Benchmark Results:**27 28| RAG Mode | Latency | Tokens | Sources | Accuracy | Cost |29|----------|---------|--------|---------|----------|------|30| Simple | 620ms | 380 | 1 | 100% | $0.001 |31| Agentic | 1800ms | 720 | 3 | 100% | $0.003 |32| Graph | 950ms | 520 | 2 | 100% | $0.002 |33 34**Winner:** Simple RAG (fastest, cheapest)35 36```json37{38  "query": "What is the company founded year?",39  "expected_answer": "2020",40  "results": {41    "simple_rag": {42      "answer": "The company was founded in 2020.",43      "latency_ms": 620,44      "tokens": {"input": 280, "output": 100},45      "sources_used": 1,46      "accuracy": 1.0,47      "cost_usd": 0.00148    },49    "agentic_rag": {50      "answer": "According to multiple sources, the company was established in 2020 with a mission to revolutionize AI education.",51      "latency_ms": 1800,52      "tokens": {"input": 520, "output": 200},53      "sources_used": 3,54      "accuracy": 1.0,55      "cost_usd": 0.00356    },57    "graph_rag": {58      "answer": "The company (Entity: CompanyX) was founded in 2020 (Relationship: founded_year).",59      "latency_ms": 950,60      "tokens": {"input": 380, "output": 140},61      "sources_used": 2,62      "accuracy": 1.0,63      "cost_usd": 0.00264    }65  }66}67```68 69---70 71### Query Category 2: Multi-Document Synthesis72 73#### Query 2.1: "Summarize company values and how they relate to technical decisions"74 75**Benchmark Results:**76 77| RAG Mode | Latency | Tokens | Sources | Accuracy | Cost |78|----------|---------|--------|---------|----------|------|79| Simple | 1200ms | 680 | 2 | 65% | $0.002 |80| Agentic | 3200ms | 1450 | 5 | 88% | $0.005 |81| Graph | 1800ms | 920 | 4 | 82% | $0.003 |82 83**Winner:** Agentic RAG (best accuracy for synthesis)84 85```json86{87  "query": "Summarize company values and how they relate to technical decisions",88  "expected_answer": "Company values: Innovation, Excellence, Community, Accessibility. These directly influence microservices architecture for scalability and accessibility",89  "results": {90    "simple_rag": {91      "answer": "Company values include Innovation, Excellence, Community, and Accessibility. The company uses microservices.",92      "latency_ms": 1200,93      "tokens": {"input": 450, "output": 230},94      "sources_used": 2,95      "accuracy": 0.65,96      "cost_usd": 0.002,97      "explanation": "Missing connection between values and architectural decisions"98    },99    "agentic_rag": {100      "answer": "The company's core values of Innovation and Accessibility drive their technical architecture decisions. Specifically: (1) Innovation enables microservices for continuous deployment. (2) Accessibility drives open API design for integration. (3) Excellence requires Kubernetes orchestration. These values manifest as React + FastAPI + PostgreSQL stack supporting 50,000+ students.",101      "latency_ms": 3200,102      "tokens": {"input": 980, "output": 470},103      "sources_used": 5,104      "accuracy": 0.88,105      "cost_usd": 0.005,106      "explanation": "Successfully linked values to architectural decisions through reasoning"107    },108    "graph_rag": {109      "answer": "Company values (Innovation, Excellence, Community, Accessibility) are implemented through: Architecture (microservices for Innovation), Tech Stack (React/FastAPI for Accessibility), and Deployment (Kubernetes for Excellence). Knowledge graph shows: Value→ArchitectureDecision with strength 0.9.",110      "latency_ms": 1800,111      "tokens": {"input": 620, "output": 300},112      "sources_used": 4,113      "accuracy": 0.82,114      "cost_usd": 0.003,115      "explanation": "Graph relationships helped identify value-architecture connections"116    }117  }118}119```120 121---122 123### Query Category 3: Complex Reasoning124 125#### Query 3.1: "Based on the technical architecture and success metrics, what would be the bottleneck if we 3x user base?"126 127**Benchmark Results:**128 129| RAG Mode | Latency | Tokens | Sources | Accuracy | Cost |130|----------|---------|--------|---------|----------|------|131| Simple | 1500ms | 850 | 2 | 52% | $0.002 |132| Agentic | 3800ms | 1680 | 6 | 91% | $0.006 |133| Graph | 2100ms | 1100 | 4 | 78% | $0.004 |134 135**Winner:** Agentic RAG (best reasoning)136 137```json138{139  "query": "Based on the technical architecture and success metrics, what would be the bottleneck if we 3x user base?",140  "expected_answer": "Database (PostgreSQL) would be bottleneck. Current: 50k students. 3x = 150k. PostgreSQL scaling requires: Read replicas, sharding, caching optimization",141  "results": {142    "simple_rag": {143      "answer": "The company uses PostgreSQL and has 50,000 students. If tripled to 150,000, there could be performance issues.",144      "latency_ms": 1500,145      "tokens": {"input": 580, "output": 270},146      "sources_used": 2,147      "accuracy": 0.52,148      "cost_usd": 0.002,149      "explanation": "Lacks depth and specific bottleneck analysis"150    },151    "agentic_rag": {152      "answer": "Analysis across multiple sources identifies database bottleneck. Current metrics: 50k students with PostgreSQL. Scaling to 150k requires: (1) Database: Implement read replicas for assessment queries (high load), (2) Caching: Redis optimization for user sessions (current cache hit: 80%), (3) Architecture: Vertical partitioning by service (User, Content, Assessment separate schemas). Critical path: PostgreSQL connections from 100→300, requiring pgBouncer setup. Estimated impact: 40% latency increase without mitigation.",153      "latency_ms": 3800,154      "tokens": {"input": 1150, "output": 530},155      "sources_used": 6,156      "accuracy": 0.91,157      "cost_usd": 0.006,158      "explanation": "Multi-step reasoning identified specific bottleneck and mitigation strategies"159    },160    "graph_rag": {161      "answer": "Knowledge graph relationships show: User(50k)→Database(PostgreSQL)→Bottleneck. Scaling path: Users(3x)→DB Load(3x)→Connection Pool(100→300)→Bottleneck. Graph identifies critical path: Assessment Service (heaviest load) → PostgreSQL → Connection limits. Recommendations: Read replicas, connection pooling tuning, sharding by course_id.",162      "latency_ms": 2100,163      "tokens": {"input": 780, "output": 320},164      "sources_used": 4,165      "accuracy": 0.78,166      "cost_usd": 0.004,167      "explanation": "Graph relationships helped identify bottleneck path"168    }169  }170}171```172 173---174 175## 📈 Aggregated Statistics176 177### By RAG Mode178 179#### Simple RAG180```181Average Latency: 1100ms182Latency Range: 600-1500ms183Average Tokens: 630/query184Average Sources: 2.1185Average Accuracy: 72%186Average Cost: $0.0018/query187Best For: Real-time, simple queries188Worst For: Complex reasoning189```190 191#### Agentic RAG192```193Average Latency: 2933ms194Latency Range: 1800-3800ms195Average Tokens: 1170/query196Average Sources: 4.7197Average Accuracy: 89%198Average Cost: $0.0045/query199Best For: Complex reasoning, multi-step200Worst For: High-throughput scenarios201```202 203#### Graph RAG204```205Average Latency: 1617ms206Latency Range: 950-2100ms207Average Tokens: 820/query208Average Sources: 3.3209Average Accuracy: 80%210Average Cost: $0.0030/query211Best For: Entity relationships, knowledge extraction212Worst For: Simple fact lookup213```214 215---216 217## 💰 Cost Analysis218 219### Per Query Costs220```221Simple RAG:   $0.0018 (baseline)222Graph RAG:    $0.0030 (1.7x)223Agentic RAG:  $0.0045 (2.5x)224```225 226### Monthly Cost (10,000 queries)227```228Simple RAG:    $18229Graph RAG:     $30230Agentic RAG:   $45231```232 233### Annual Cost (120,000 queries)234```235Simple RAG:    $216236Graph RAG:     $360237Agentic RAG:   $540238```239 240---241 242## 📊 Performance by Model243 244### Llama 3.1 8B (Fast)245```246Simple RAG:   920ms avg247Agentic RAG: 2100ms avg248Graph RAG:   1450ms avg249```250 251### Llama 3.3 70B (Quality)252```253Simple RAG:   1200ms avg254Agentic RAG: 3100ms avg255Graph RAG:   1850ms avg256(Better accuracy, 30% slower)257```258 259### GPT-OSS 120B (Enterprise)260```261Simple RAG:   1450ms avg262Agentic RAG: 3500ms avg263Graph RAG:   2200ms avg264(Best quality, 50% slower)265```266 267---268 269## 🎯 Recommendations Matrix270 271### Use Simple RAG When:272✅ Response time < 1 second required273✅ Budget-conscious ($15-20/month)274✅ Simple fact lookups275✅ High throughput needed (>1000 qps)276✅ Single-document queries277 278### Use Agentic RAG When:279✅ Accuracy > 85% required280✅ Multi-step reasoning needed281✅ Complex document synthesis282✅ Tool use / sub-queries needed283✅ Expert systems284 285### Use Graph RAG When:286✅ Entity relationships important287✅ Knowledge extraction critical288✅ Balanced latency/accuracy (1-2s)289✅ Domain expertise required290✅ Complex document linking291 292---293 294## 📥 CSV Export Format295 296```csv297query_id,query_text,rag_mode,latency_ms,input_tokens,output_tokens,sources_used,accuracy,cost_usd,model_used,timestamp298 2991,"What is the company founded year?",simple,620,280,100,1,1.0,0.001,"llama-3.1-8b",2026-06-25T10:00:00Z3001,"What is the company founded year?",agentic,1800,520,200,3,1.0,0.003,"llama-3.1-8b",2026-06-25T10:00:05Z3011,"What is the company founded year?",graph,950,380,140,2,1.0,0.002,"llama-3.1-8b",2026-06-25T10:00:08Z302```303 304---305 306## 🔗 Benchmark Files307 308All benchmark data is available in:309- `data/benchmark_results.json` - Raw JSON310- `data/benchmark_results.csv` - CSV format311- `rag_comparison_report.html` - Interactive HTML report312- `BENCHMARK_RESULTS.md` - Detailed markdown313 314---315 316## 🚀 Using This Data317 318### For Research319```python320import json321with open('data/benchmark_results.json') as f:322    data = json.load(f)323# Analyze performance metrics324# Compare across modes and models325```326 327### For Visualization328```python329import pandas as pd330df = pd.read_csv('data/benchmark_results.csv')331# Create custom visualizations332# Export to Excel / Power BI333```334 335### For Deployment Decisions336- Use decision matrix above337- Consider your specific SLA requirements338- Benchmark with your own documents339- Test with your expected query patterns340 341---342 343**Note:** All data is sample/representative. Actual performance depends on:344- Document size and complexity345- Query complexity346- Model selection347- Hardware/network conditions348- Concurrent load349 350For accurate metrics: Run `python benchmark.py` with your own documents.351