onyx-dot-app/EnterpriseRAG-Bench-Leaderboard

Note: The YAML frontmatter above (between the --- markers) is Hugging Face Spaces metadata. It configures the Space card (emoji, banner colors, SDK, etc.) and must not be removed or the HF Space deployment will break.

EnterpriseRAG-Bench Leaderboard

A Gradio-based leaderboard app for EnterpriseRAG-Bench, a RAG benchmark for company internal knowledge. Hosted on HuggingFace Spaces.

Where This Code Lives

This project has two remotes:

• —GitHub (private): onyx-dot-app/EnterpriseRAG-Bench-Leaderboard-Private — used for collaboration and in-progress work between releases.
• —Hugging Face Spaces: onyx-dot-app/EnterpriseRAG-Bench-Leaderboard — pushing to this remote automatically deploys the app.

The typical workflow is to develop and review changes on the private GitHub repo, then push to the Hugging Face remote when ready to deploy.

Repository Structure

├── app.py                        # Entry point
├── create_leaderboard.py         # Gradio app definition
├── transform_raw_data.py         # Generates display data from raw evaluation files
├── requirements.txt
├── tabs/
│   ├── shared_data.py            # Shared data loading and caching
│   ├── leaderboard_tab.py        # Main ranked leaderboard table
│   ├── submission_overview_tab.py # Per-system breakdown by question type
│   ├── data_viewer_tab.py        # Browse questions, gold answers, and facts
│   ├── submission_viewer_tab.py  # Single system per-question viewer
│   └── data_viewer_side_by_side_tab.py  # Compare two systems side-by-side
└── data/
    ├── raw_data/                 # Source evaluation files
    │   ├── systems.yaml          # System key -> display name mapping
    │   ├── questions.jsonl       # Base question set (500 questions)
    │   ├── answers_<key>.jsonl   # Per-system answers
    │   ├── results_<key>.json    # Per-system evaluation results
    │   └── questions_updated_<key>.jsonl  # Per-system corrected questions
    └── final_display_data/       # Generated files (do not edit directly)
        ├── leaderboard.csv
        └── data_viewer.jsonl

Running Locally

pip install -r requirements.txt
python app.py

The app will be available at http://localhost:7860.

Adding a New System

1. 1.Add raw data files to data/raw_data/:
2. 2.answers_<key>.jsonl — one JSON object per line with question_id and answer
3. 3.results_<key>.json — evaluation results with aggregate_stats and per-question questions array
4. 4.questions_updated_<key>.jsonl — corrected question set (output of the evaluation pipeline)

The <key> is a short identifier that must be consistent across all three files. Naming convention: lowercase ASCII, with underscores (_) as the only separator — no hyphens, no spaces. Multi-word system names use underscores between tokens (e.g., openai_filesearch, weaviate_verba, amazon_q_kendra).

Currently registered keys: agent, amazon_q_kendra, anythingllm, bm25, langchain, llamaindex_semantic_k2, nvidia_ai_blueprints, openai_filesearch, openclaw, owui_chroma, ragflow, vector, vertexai, weaviate_verba.

1. 1.Add a display name in data/raw_data/systems.yaml:

   your_key: "Your System Display Name"

1. 1.Regenerate display data:

   python transform_raw_data.py

The script auto-discovers systems from results_*.json files and validates that all required files and mappings exist. It will error on missing fields, malformed JSON, or unmapped system keys.

1. 1.Validate that every question is fully populated before publishing. A submission is only ready for the scoreboard once all 500 questions are present and valid for the new system — every record must have a non-empty answer, a populated correctness / completeness, and the same question ID set as the other systems. Recall metrics (recall, invalid_extra_docs) are expected to be null only on high_level and info_not_found question types; everywhere else they must be populated. Do not push a partial run. Quick sanity check:

   python3 -c "
   import json
   from collections import Counter
   c = Counter()
   with open('data/final_display_data/data_viewer.jsonl') as f:
       for line in f:
           c[json.loads(line)['model_name']] += 1
   for m, n in sorted(c.items()):
       print(f'{m}: {n}')   # every system should report 500
   "

1. 1.Preview locally with python app.py, then push when ready.

Updating Existing Data

To refresh with updated evaluation files:

1. 1.Replace the relevant files in data/raw_data/ (or replace all of them).
2. 2.Run python transform_raw_data.py to regenerate data/final_display_data/.
3. 3.Restart the app to pick up changes (the data is cached in memory on first load).