shuklaved/eris-retrieval-benchmark

Eris GPU-Accelerated Semantic Retrieval Challenge

Welcome to the Eris GPU-Accelerated Semantic Retrieval Challenge platform. This repository contains the complete benchmark dataset, baseline implementations, evaluation grading infrastructure, and reference solution.

1. Dataset Overview

The Eris Challenge evaluates high-performance semantic retrieval models over scientific literature abstracts derived from SciFact.

Benchmark Specifications

• —Canonical Public Dataset Structure: public_dataset/documents.csv (500 abstracts) and public_dataset/queries.csv (120 queries).
• —Queries & Splits: 120 queries across Train (86), Validation (17), and Test (17).
• —Candidate Pool Relevance: 1,560 judged query-document pairs (relevance.csv), with exactly 13 candidate documents per query.
• —Evaluation Metric: Candidate-Pool NDCG@10 (Primary) alongside Hit@1, Hit@5, and Hit@10.

[!IMPORTANT] Candidate-Pool NDCG@10 Methodology Definition: Evaluation is conducted strictly over the 13 judged candidate documents per query rather than exhaustive corpus-wide judgments. NDCG@10 measures candidate ranking quality within the judged pool and must not be interpreted as an unbiased estimate of exhaustive-corpus NDCG.

2. Directory Structure

.
├── challenge/
│   ├── problem_description.md   # Comprehensive dataset description & challenge rules
│   ├── rubrics.md               # Grading rubric & evaluation criteria
│   └── config.yaml              # Global benchmark configuration
├── public_dataset/              # Canonical public dataset files
│   ├── documents.csv            # 500 public biomedical research abstracts
│   └── queries.csv              # 120 public evaluation queries (with candidate pool IDs)
├── data/
│   └── prototype/               # Internal prototype source data & hidden ground truth
│       ├── documents_prototype_500.csv
│       ├── queries.csv
│       └── relevance.csv        # Private ground-truth judgments (1,560 pairs)
├── grading/
│   ├── grade.py                 # Automated challenge grader script
│   └── evaluation_results.json  # Exported evaluation metrics output
├── solution/
│   ├── inference.py             # Reference solution inference pipeline
│   ├── evaluate_local.py        # Local evaluation runner
│   ├── train.py                 # Reference solution training entrypoint
│   ├── requirements.txt         # Dependencies
│   └── README.md                # Solution usage documentation
├── src/
│   └── retrieval/
│       ├── tfidf_baseline.py    # Stage 5 CPU TF-IDF baseline
│       └── semantic_baseline.py # Stage 6 Dense Semantic baseline
├── experiments/
│   ├── stage5/                  # Stage 5 baseline outputs & metrics
│   ├── stage6/                  # Stage 6 baseline outputs & metrics
│   ├── stage7_8/                # Stage 7-8 audit reports & stats
│   └── final_stage_report.md    # Final engineering report
├── eris_compliance_report.md    # Dataset quality compliance audit report
└── README.md                    # Root project documentation

3. Baseline Performance Comparison

Performance Breakdown by Query Category

4. Hardware & Execution Device Notice

• —Target hardware: NVIDIA GeForce GTX 1650 4GB VRAM.
• —In environments where PyTorch CUDA support is present, inference.py automatically uses cuda:0.
• —In CPU-only PyTorch environments (such as Python 3.14 on Windows), inference.py transparently falls back to CPU execution while retaining full algorithmic compatibility.

5. Quick Start Instructions

Running the Reference Solution

python solution/inference.py

Running Local Evaluation & Challenge Grader

python solution/evaluate_local.py

6. License & Source Attribution

• —Upstream Dataset: SciFact (Wadden et al., EMNLP 2020)
• —Source URL: https://huggingface.co/datasets/mteb/scifact
• —License: CC-BY-4.0 (Creative Commons Attribution 4.0 International)