Apppublic

Adityax-07/CodeSage

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

App README

<img width="100%" src="https://capsule-render.vercel.app/api?type=waving&color=gradient&customColorList=6,11,20&height=200&section=header&text=CodeSage%20🧙&fontSize=60&fontColor=fff&animation=twinkling&fontAlignY=35&desc=LLM%20vs%20RAG%20vs%20Fine-Tuning%20—%20Live%20Trade-off%20Platform&descAlignY=60&descSize=18" />

<img src="https://readme-typing-svg.demolab.com?font=Fira+Code&weight=700&size=22&pause=1200&color=A78BFA&center=true&vCenter=true&multiline=true&repeat=true&width=700&height=60&lines=Same+Question.+Three+Architectures.+Real+Numbers." alt="Typing SVG" />

<a href="https://github.com/Adityax-07/LLM-vs-RAG-vs-Fine-Tuning-/stargazers"> <img src="https://img.shields.io/github/stars/Adityax-07/LLM-vs-RAG-vs-Fine-Tuning-?style=for-the-badge&logo=starship&color=f59e0b&logoColor=white&labelColor=1a1a2e" /> </a> <a href="https://github.com/Adityax-07/LLM-vs-RAG-vs-Fine-Tuning-/network/members"> <img src="https://img.shields.io/github/forks/Adityax-07/LLM-vs-RAG-vs-Fine-Tuning-?style=for-the-badge&logo=git&color=8b5cf6&logoColor=white&labelColor=1a1a2e" /> </a> <img src="https://img.shields.io/badge/License-MIT-22c55e?style=for-the-badge&logo=opensourceinitiative&logoColor=white&labelColor=1a1a2e" /> <img src="https://img.shields.io/badge/Status-Production%20Ready-22c55e?style=for-the-badge&logo=checkmarx&logoColor=white&labelColor=1a1a2e" /> <img src="https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&logo=python&logoColor=white&labelColor=1a1a2e" />

<a href="https://huggingface.co/spaces/Adityax-07/CodeSage"> <img src="https://img.shields.io/badge/🤗%20Live%20Demo-Try%20CodeSage%20on%20HF%20Spaces-FFD21E?style=for-the-badge&labelColor=1a1a2e" /> </a>

<img src="https://img.shields.io/badge/Streamlit-FF4B4B?style=for-the-badge&logo=streamlit&logoColor=white" /> <img src="https://img.shields.io/badge/LangChain-1C3C3C?style=for-the-badge&logo=langchain&logoColor=white" /> <img src="https://img.shields.io/badge/GroqAPI-F55036?style=for-the-badge&logo=groq&logoColor=white" /> <img src="https://img.shields.io/badge/HuggingFace-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black" /> <img src="https://img.shields.io/badge/FAISS-0467DF?style=for-the-badge&logo=meta&logoColor=white" /> <img src="https://img.shields.io/badge/LoRA%2FPEFT-EF4444?style=for-the-badge&logo=pytorch&logoColor=white" /> <img src="https://img.shields.io/badge/Plotly-3F4F75?style=for-the-badge&logo=plotly&logoColor=white" /> <img src="https://img.shields.io/badge/GoogleColab-F9AB00?style=for-the-badge&logo=googlecolab&logoColor=black" />

<img src="https://skillicons.dev/icons?i=python,pytorch,tensorflow,git,github,vscode&theme=dark" />

<blockquote> 🧪 CodeSage is a live, side-by-side AI research platform that fires the same programming question at three fundamentally different architectures — Baseline LLM, RAG, and Fine-Tuning — then auto-scores every answer on accuracy, hallucination, groundedness, relevance, and cost. No cherry-picking. No manual grading. Real numbers, real trade-offs. </blockquote>

<img src="https://img.shields.io/badge/50-Benchmark%20Questions-8b5cf6?style=flat-square" /> <img src="https://img.shields.io/badge/3-AI%20Systems%20Compared-06b6d4?style=flat-square" /> <img src="https://img.shields.io/badge/8-Auto%20Eval%20Metrics-f59e0b?style=flat-square" /> <img src="https://img.shields.io/badge/85.3%25-Fine--Tune%20Accuracy-22c55e?style=flat-square" /> <img src="https://img.shields.io/badge/0%25-Hallucination%20Rate-ef4444?style=flat-square" />

</div>

📌 Table of Contents

	Section
⚡	Benchmark Results
🧠	What is CodeSage?
✨	Features
🏗️	Architecture
📊	Evaluation Pipeline
🚀	Quick Start
📚	Knowledge Base
💡	Decision Guide
🛠️	Tech Stack
🗂️	Project Structure
🔮	Roadmap

⚡ Benchmark Results

Full evaluation: 3 systems × 50 Q&A pairs × 8 metrics — fully automated, zero manual grading

📏 Metric	🔵 Baseline LLM	🟢 RAG Chatbot	🟣 Fine-Tuned (Qwen2.5 + LoRA)
🎯 Answer Accuracy	61.4%	81.6%	85.3% ✨
🚫 Hallucination Rate	43.2% ❌	9.8%	0.0% ✨
🔍 Answer Relevance	0.714	0.768	0.891 ✨
📌 Groundedness	—	0.87 ✨	—
⚡ Avg Latency	~1.2s	~2.1s	~0.4s ✨
💰 Cost / Query	~$0.0020	~$0.0030	$0.0002 ✨

🔑 Key Findings

Insight	Detail
🚫 Hallucination gap	Baseline hallucinates on `43.2%` of questions — Fine-Tuning eliminates this entirely → `0%`
📉 RAG cuts hallucination 4.4×	From `43.2%` → `9.8%` purely through grounded retrieval, no retraining needed
💰 Fine-Tuning is 10× cheaper	`$0.0002` vs `~$0.002` per query — smaller model, fully local inference
⚡ Fine-Tuning is 3× faster	`0.4s` vs `1.2s` — no retrieval pipeline, no large-model API round-trip
🎯 No universal winner	RAG wins on updatability · Fine-Tuning wins on cost/speed/precision · Baseline wins on zero-setup

🧠 What is CodeSage?

CodeSage is a decision-making tool for AI engineers. When building a domain-specific assistant, you always hit the same three-way fork:

              ┌──────────────────────────────────────────┐
              │       Domain-Specific AI Assistant        │
              └────────────────────┬─────────────────────┘
                                   │
         ┌─────────────────────────┼─────────────────────────┐
         ▼                         ▼                         ▼
┌─────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  BASELINE LLM   │     │   RAG PIPELINE   │     │   FINE-TUNING    │
│                 │     │                  │     │                  │
│ + Zero setup    │     │ + Always fresh   │     │ + 10x cheaper    │
│ + Broad topics  │     │ + Grounded       │     │ + 0% hallucin.   │
│ - Hallucinates  │     │ - Retrieval lag  │     │ - Hard to update │
└─────────────────┘     └──────────────────┘     └──────────────────┘

CodeSage makes this trade-off visible and measurable — same question, same moment, real output from all three.

✨ Features

<div align="center"> <table> <tr> <td align="center" width="220"> 🔀 Side-by-Side Compare Three answers to one question, simultaneously, in one view </td> <td align="center" width="220"> 📊 Auto Evaluation 8-metric LLM-as-Judge scores every response automatically </td> <td align="center" width="220"> 🏆 Winner Badge Best answer highlighted; hallucination flag raised on low-confidence </td> </tr> <tr> <td align="center"> 📈 Analytics Dashboard Plotly charts + paper-style TABLE II aggregated over 50 benchmarks </td> <td align="center"> 💾 Persistent Cache Results stored in <code>benchmark_cache.json</code> — instant reload, no re-running </td> <td align="center"> 📄 PDF Ingestion Drop any PDF into <code>data/pdfs/</code> — RAG ingests it automatically </td> </tr> </table> </div>

🏗️ Architecture

╔══════════════════════════════════════════════════════════════════════════╗
║                           🖥️  Streamlit UI                               ║
║  ┌──────────────────┐   ┌──────────────────────┐   ┌──────────────────┐ ║
║  │  ⚡ System 1      │   │   🔍 System 2         │   │  🧠 System 3     │ ║
║  │  Baseline LLM    │   │   RAG Pipeline        │   │  Fine-Tuned      │ ║
║  └────────┬─────────┘   └──────────┬────────────┘   └────────┬─────────┘ ║
╚═══════════╪═══════════════════════╪════════════════════════╪═════════════╝
            │                       │                         │
            ▼                       ▼                         ▼
     ┌─────────────┐       ┌──────────────────┐      ┌───────────────────┐
     │  Groq API   │       │   FAISS Index    │      │  Qwen2.5-1.5B     │
     │ Llama-3.1-8B│       │ all-MiniLM-L6-v2 │      │  + LoRA Adapters  │
     │ (zero-shot) │       │  (top-3 chunks)  │      │  (PEFT, local)    │
     └─────────────┘       └────────┬─────────┘      └───────────────────┘
                                    │
                             Groq API (with context)
                                    │
                          ┌─────────▼──────────┐
                          │   🏛️  LLM-as-Judge  │
                          │  8 metrics, auto    │
                          └────────────────────┘

⚡ System 1 — Baseline LLM

Sends the question directly to Llama-3.1-8B via Groq with a minimal system prompt. No extra knowledge. Represents what an off-the-shelf LLM can do — the floor every other system must beat.

🔍 System 2 — RAG Pipeline

1.Question → all-MiniLM-L6-v2 embedding
2.Top-3 chunks retrieved from FAISS vector store (17 documents)
3.Chunks injected as context into Llama-3.1-8B via Groq
4.Groundedness scored — answers must be traceable to retrieved text

🧠 System 3 — Fine-Tuned Model

Qwen2.5-1.5B fine-tuned with LoRA (r=8, α=32) on curated CS Q&A pairs via Google Colab T4 GPU. Adapters loaded locally via peft — zero cloud inference cost, sub-second latency.

📊 Evaluation Pipeline

Each answer is auto-scored by an LLM judge across 8 dimensions:

Icon	Metric	Description	Unit
🎯	Answer Accuracy	Cosine similarity of answer vs reference embedding	%
📌	Groundedness	Cosine similarity of answer vs retrieved context	0–1
🚫	Hallucination Rate	% of answers with accuracy < 0.5	%
🔍	Answer Relevance	Cosine similarity of answer vs question	0–1
📜	Faithfulness (ROUGE-L)	Token overlap with source context or reference	0–1
⏱️	Avg Response Time	Mean latency per query	sec
💰	Cost per Query	Token-count-based cost estimate	USD
⭐	Overall Score	30% Acc + 20% Ground + 20% (1−HR) + 15% Rel + 15% Faith	1–5

🚀 Quick Start

`Step 1` — Clone & Install

bash

git clone https://github.com/Adityax-07/LLM-vs-RAG-vs-Fine-Tuning-.git
cd LLM-vs-RAG-vs-Fine-Tuning-
pip install -r requirements.txt

`Step 2` — Configure API Key

bash

echo "GROQ_API_KEY=your_key_here" > .env

🆓 Free key at console.groq.com

`Step 3` — Launch

bash

streamlit run demo.py

FAISS vector store builds automatically on first launch. Systems 1 & 2 are ready instantly.

`Step 4` — (Optional) Activate Fine-Tuned Model

bash

python -c "
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen2.5-1.5B-Instruct')
model = PeftModel.from_pretrained(base, 'checkpoint-25')
model.merge_and_unload().save_pretrained('finetuned_model')
AutoTokenizer.from_pretrained('checkpoint-25').save_pretrained('finetuned_model')
"

Or open system3_finetune_colab.ipynb in Google Colab to train from scratch on a free T4 GPU (~10 min).

`Step 5` — Regenerate Benchmark (optional)

bash

# Pre-computed results already included in data/benchmark_cache.json
python run_benchmark.py

📚 Knowledge Base

The RAG system retrieves from 17 hand-crafted topic documents in data/docs/:

<div align="center"> <table> <tr> <td valign="top" width="33%"> 🧮 Algorithms & DSA <code>binarysearch</code> <code>sortingalgorithms</code> <code>dynamicprogramming</code> <code>graphalgorithms</code> <code>trees</code> <code>linkedlist</code> <code>stackqueue</code> <code>recursion</code> <code>backtracking</code> </td> <td valign="top" width="33%"> 📐 More DSA <code>greedyalgorithms</code> <code>hashing</code> <code>stringalgorithms</code> <code>twopointers</code> <code>bigonotation</code> <code>heaps</code> </td> <td valign="top" width="33%"> 🌐 Web & Tooling <code>reacthooks</code> <code>restapi</code> <code>javascriptpromises</code> <code>cssflexbox</code> <code>typescriptbasics</code> <code>sqlbasics</code> <code>gitbasics</code> </td> </tr> </table> </div>

💡 Decision Guide

🤔 Situation	✅ Best Choice	📝 Why
Prototyping or general queries	Baseline LLM	Zero setup, covers broad topics well
Knowledge changes frequently	RAG	Update docs without retraining
Fixed domain, cost/latency matters	Fine-Tuning	10× cheaper, 3× faster, 0% hallucination
Need citations & traceability	RAG	Groundedness score + visible source chunks
Production with tight latency SLA	Fine-Tuning	Local inference, no API round-trip

🛠️ Tech Stack

<div align="center"> <table> <tr> <th>Layer</th> <th>Technology</th> <th>Purpose</th> </tr> <tr> <td>📊 UI</td> <td> <img src="https://img.shields.io/badge/Streamlit-FF4B4B?style=flat-square&logo=streamlit&logoColor=white" /> <img src="https://img.shields.io/badge/Plotly-3F4F75?style=flat-square&logo=plotly&logoColor=white" /> </td> <td>3-way comparison dashboard + analytics charts</td> </tr> <tr> <td>⚡ LLM</td> <td><img src="https://img.shields.io/badge/GroqAPI-F55036?style=flat-square&logo=groq&logoColor=white" /></td> <td>Llama-3.1-8B — Baseline + RAG generation</td> </tr> <tr> <td>🤖 Embeddings</td> <td><img src="https://img.shields.io/badge/sentence--transformers-FFD21E?style=flat-square&logo=huggingface&logoColor=black" /></td> <td><code>all-MiniLM-L6-v2</code> — RAG semantic retrieval</td> </tr> <tr> <td>🔍 Vector DB</td> <td><img src="https://img.shields.io/badge/FAISS-0467DF?style=flat-square&logo=meta&logoColor=white" /></td> <td>CPU-based semantic search over knowledge base</td> </tr> <tr> <td>🧠 Fine-Tuning</td> <td> <img src="https://img.shields.io/badge/PEFT%2FLoRA-EF4444?style=flat-square&logo=pytorch&logoColor=white" /> <img src="https://img.shields.io/badge/Transformers-FFD21E?style=flat-square&logo=huggingface&logoColor=black" /> </td> <td>LoRA adapter (r=8, α=32) on Qwen2.5-1.5B</td> </tr> <tr> <td>🏋️ Base Model</td> <td><img src="https://img.shields.io/badge/Qwen2.5--1.5B-FFD21E?style=flat-square&logo=huggingface&logoColor=black" /></td> <td>Alibaba's compact LLM — LoRA fine-tuned locally</td> </tr> <tr> <td>☁️ Training</td> <td><img src="https://img.shields.io/badge/GoogleColab-F9AB00?style=flat-square&logo=googlecolab&logoColor=black" /></td> <td>Free T4 GPU — LoRA training in ~10 minutes</td> </tr> <tr> <td>🔗 Orchestration</td> <td><img src="https://img.shields.io/badge/LangChain-1C3C3C?style=flat-square&logo=langchain&logoColor=white" /></td> <td>RAG pipeline, FAISS integration, PDF ingestion</td> </tr> <tr> <td>📏 Metrics</td> <td><img src="https://img.shields.io/badge/rouge--score-EF4444?style=flat-square&logo=python&logoColor=white" /></td> <td>ROUGE-L + cosine similarity for auto-evaluation</td> </tr> </table> </div>

🗂️ Project Structure

📦 LLM-vs-RAG-vs-Fine-Tuning/
│
├── 📄 demo.py                         ← Streamlit app (main entry point)
├── 📄 system1_baseline.py             ← Baseline LLM via Groq API
├── 📄 system2_rag.py                  ← RAG pipeline: FAISS + LangChain + Groq
├── 📄 system3_inference.py            ← Fine-tuned model inference (PEFT)
├── 📓 system3_finetune_colab.ipynb    ← LoRA training notebook (Colab T4)
├── 📄 evaluate.py                     ← Standalone evaluation script
├── 📄 run_benchmark.py                ← Regenerates benchmark_cache.json
│
├── 📁 checkpoint-25/                  ← Trained LoRA weights (included)
│   ├── adapter_model.safetensors
│   ├── adapter_config.json            ← r=8, alpha=32
│   └── tokenizer.json
│
├── 📁 finetuned_model/                ← Merged model (after merge step)
│
├── 📁 data/
│   ├── 📁 docs/                       ← 17 knowledge base .txt files
│   ├── 📁 faiss_index/                ← FAISS vector store (auto-built)
│   ├── 📁 pdfs/                       ← Drop PDFs here for RAG ingestion
│   ├── benchmark_cache.json           ← Pre-computed 50Q benchmark results
│   ├── reference_answers.json         ← Ground-truth Q&A pairs
│   └── finetune_data.jsonl            ← LoRA training data (ChatML format)
│
└── 📄 requirements.txt

🔮 Roadmap

Status	Feature
✅	50-question auto-benchmark with persistent cache
✅	LoRA fine-tune checkpoint (`checkpoint-25`) included
✅	Analytics dashboard with Plotly + TABLE II
✅	PDF ingestion into RAG knowledge base
🔜	Push Qwen2.5 LoRA adapter to HuggingFace Hub
🔜	Full 3-system live demo on HuggingFace Spaces
🔜	Expand knowledge base: 17 → 50+ documents
🔜	RAGAS-style faithfulness + context precision metrics
🔜	Custom knowledge base upload via Streamlit UI

<img width="100%" src="https://capsule-render.vercel.app/api?type=waving&color=gradient&customColorList=6,11,20&height=120&section=footer" />

Built with 🧠 by <a href="https://github.com/Adityax-07">Adityax-07</a>

Powered by Groq · HuggingFace · FAISS · LangChain · Streamlit

⭐ If CodeSage helped you understand the LLM trade-off space, drop a star!

</div>

📌 Table of Contents

⚡ Benchmark Results

🔑 Key Findings

🧠 What is CodeSage?

✨ Features

🏗️ Architecture

⚡ System 1 — Baseline LLM

🔍 System 2 — RAG Pipeline

🧠 System 3 — Fine-Tuned Model

📊 Evaluation Pipeline

🚀 Quick Start

Step 1 — Clone & Install

Step 2 — Configure API Key

Step 3 — Launch

Step 4 — (Optional) Activate Fine-Tuned Model

Step 5 — Regenerate Benchmark (optional)

📚 Knowledge Base

💡 Decision Guide

🛠️ Tech Stack

🗂️ Project Structure

🔮 Roadmap

`Step 1` — Clone & Install

`Step 2` — Configure API Key

`Step 3` — Launch

`Step 4` — (Optional) Activate Fine-Tuned Model

`Step 5` — Regenerate Benchmark (optional)