yugandhartamire/rag_bench_pipeline

🚀 RAG Pipeline with TRACe Evaluation

A complete Retrieval-Augmented Generation (RAG) system with comprehensive evaluation using TRACe metrics (Relevance, Utilization, Completeness, Adherence). Built with Gradio for easy deployment to Hugging Face Spaces.

✨ Features

• —🔍 Multiple Retrieval Methods: Dense (vector similarity) and Sparse (BM25)
• —🎯 Advanced Reranking: FlashRank with multiple repacking strategies
• —🤖 Multiple LLM Providers: Groq and OpenRouter support
• —📊 TRACe Metrics: Complete evaluation with Relevance, Utilization, Completeness, and Adherence
• —🎨 Gradio Interface: User-friendly web UI with multiple tabs
• —📈 Batch Processing: Process multiple queries with checkpoint support
• —🔄 API Key Rotation: Automatic rate limit handling
• —💾 Results Export: Download evaluation results as CSV

🏗️ Architecture

rag_pipeline/
├── app/
│   ├── config.py              # Configuration management
│   ├── data_loader.py         # Dataset loading utilities
│   ├── llm_manager.py         # LLM provider management
│   ├── rag/                   # Core RAG components
│   │   ├── embeddings.py      # HuggingFace embeddings
│   │   ├── retrievers.py      # BM25 & Vector retrievers
│   │   ├── reranker.py        # FlashRank reranking
│   │   ├── pipeline.py        # Main RAG pipeline
│   │   └── evaluator.py       # TRACe metrics evaluation
│   ├── utils/                 # Utility functions
│   │   ├── text_processing.py # Text splitting & keying
│   │   ├── metrics.py         # Metrics computation
│   │   └── helpers.py         # Error handling
│   └── ui/                    # Gradio interface
│       └── gradio_app.py      # Multi-tab UI
├── app.py                     # Application entry point
├── config.yaml                # Default configuration
├── requirements.txt           # Python dependencies
└── README.md                  # This file

🚀 Quick Start

Local Installation

1. 1.Clone the repository

git clone <your-repo-url>
cd rag_pipeline

1. 1.Create virtual environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

1. 1.Install dependencies

pip install -r requirements.txt

1. 1.Set up API keys

Create a .env file or set environment variables:

export GROQ_API_KEY="your_groq_api_key"
export OPENROUTER_API_KEY="your_openrouter_api_key"

1. 1.Run the application

python app.py

The application will be available at http://localhost:7860

🌐 Deploy to Hugging Face Spaces

Method 1: Direct Upload

1. 1.Create a new Space on Hugging Face
2. 2.Choose Gradio as the SDK
3. 3.Upload all project files
4. 4.Add your API keys in Settings → Repository secrets:
5. 5.GROQ_API_KEY
6. 6.OPENROUTER_API_KEY

Method 2: Git Integration

1. 1.Create a new Space with Git enabled
2. 2.Clone your Space repository:

git clone https://huggingface.co/spaces/<your-username>/<your-space-name>

1. 1.Copy project files and push:

cp -r rag_pipeline/* <your-space-name>/
cd <your-space-name>
git add .
git commit -m "Initial commit"
git push

📖 Usage Guide

Tab 1: Configuration ⚙️

1. 1.Dataset Configuration
2. 2.Enter Hugging Face dataset name (e.g., rungalileo/ragbench)
3. 3.Select split (train/validation/test)
4. 4.Set sample size (-1 for all)

1. 1.Chunking Settings
2. 2.Chunk size: 400 (recommended)
3. 3.Chunk overlap: 80 (recommended)

1. 1.Retrieval Configuration
2. 2.Embedding model: BAAI/bge-small-en-v1.5
3. 3.Retriever type: Dense or Sparse
4. 4.Top K: Number of documents to retrieve

1. 1.Reranking Settings
2. 2.Ranking model: ms-marco-MiniLM-L-12-v2
3. 3.Rerank K: Final number of documents
4. 4.Repacking strategy: sides or reverse

1. 1.LLM Configuration
2. 2.Provider: Groq or OpenRouter
3. 3.Generator model: llama-3.1-8b-instant
4. 4.Judge model: llama-3.3-70b-versatile

1. 1.API Keys
2. 2.Enter comma-separated API keys for rate limit handling
3. 3.Click Load Dataset & Initialize Pipeline

Tab 2: Single Query 🔍

1. 1.Enter your question
2. 2.Choose:
3. 3.Run Query: Get answer without evaluation
4. 4.Run Query + Evaluation: Full TRACe metrics evaluation
5. 5.View retrieved documents, generated answer, and metrics

Tab 3: Batch Evaluation 📊

1. 1.Set number of samples to process
2. 2.Click Start Batch Processing
3. 3.Monitor progress in real-time
4. 4.Download results as CSV

Tab 4: Results Analysis 📈

1. 1.Upload batch results CSV
2. 2.View summary statistics
3. 3.Analyze metrics distribution

📊 TRACe Metrics Explained

Relevance Score

Fraction of context sentences that are relevant to the question.

• —Formula: relevant_sentences / total_context_sentences
• —Range: 0.0 to 1.0

Utilization Score

Fraction of context sentences actually used in the response.

• —Formula: utilized_sentences / total_context_sentences
• —Range: 0.0 to 1.0

Completeness Score

How well the response covers all relevant information.

• —Formula: (relevant ∩ utilized) / relevant_sentences
• —Range: 0.0 to 1.0

Adherence Score

Whether the response is fully supported by context.

• —Boolean: True/False
• —Continuous: Fraction of supported sentences

🔧 Configuration Options

Retriever Types

Dense (Vector Similarity)

• —Uses semantic embeddings
• —Better for conceptual matches
• —Requires embedding model

Sparse (BM25)

• —Uses lexical matching
• —Better for exact term matches
• —Faster, no embedding needed

Repacking Strategies

Sides

• —Places most relevant docs at beginning and end
• —Leverages attention bias in LLMs

Reverse

• —Least to most relevant order
• —Alternative ordering strategy

🤝 API Key Management

The system supports multiple API keys for each provider to handle rate limits:

# In Configuration tab, enter comma-separated keys:
groq_keys = "key1,key2,key3"
openrouter_keys = "key1,key2"

The system automatically rotates keys when rate limits are hit.

📦 Dependencies

Key libraries:

• —gradio - Web interface
• —langchain - LLM orchestration
• —sentence-transformers - Embeddings
• —flashrank - Reranking
• —rank-bm25 - Sparse retrieval
• —datasets - HuggingFace datasets

See `requirements.txt` for complete list.

🐛 Troubleshooting

Rate Limit Errors

• —Add multiple API keys (comma-separated)
• —System will automatically rotate

Memory Issues

• —Reduce sample size
• —Use smaller embedding models
• —Reduce chunk size

Slow Processing

• —Use sparse retrieval (BM25)
• —Reduce rerank_k value
• —Use smaller models

📝 Example Datasets

Compatible with datasets from Hugging Face with this structure:

{
    "id": "unique_id",
    "question": "Your question here?",
    "documents": ["passage1", "passage2", ...],
    "dataset_name": "dataset_name"
}

Tested datasets:

• —rungalileo/ragbench (recommended)
• —Custom datasets with similar structure

🛠️ Development

Adding New Retrievers

Edit `app/rag/retrievers.py`:

class CustomRetriever:
    def invoke(self, query: str, k: int) -> List[Document]:
        # Your implementation
        pass

Adding New Metrics

Edit `app/utils/metrics.py`:

def compute_custom_metric(judge_output, context):
    # Your implementation
    return score

📄 License

MIT License - feel free to use for your projects!

🙏 Acknowledgments

• —Built with LangChain
• —Uses FlashRank for reranking
• —Evaluation based on TRACe metrics
• —Interface powered by Gradio

📧 Contact

For issues and questions, please open an issue on GitHub.

Made with ❤️ for the RAG community