open-llm-leaderboard/open_llm_leaderboard

1# Backend - Open LLM Leaderboard 🏆2 3FastAPI backend for the Open LLM Leaderboard. This service is part of a larger architecture that includes a React frontend. For complete project installation, see the [main README](../README.md).4 5## ✨ Features6 7- 📊 REST API for LLM models leaderboard management8- 🗳️ Voting and ranking system9- 🔄 HuggingFace Hub integration10- 🚀 Caching and performance optimizations11 12## 🏗 Architecture13 14```mermaid15flowchart TD16    Client(["**Frontend**<br><br>React Application"]) --> API["**API Server**<br><br>FastAPI REST Endpoints"]17 18    subgraph Backend19        API --> Core["**Core Layer**<br><br>• Middleware<br>• Cache<br>• Rate Limiting"]20        Core --> Services["**Services Layer**<br><br>• Business Logic<br>• Data Processing"]21 22        subgraph Services Layer23            Services --> Models["**Model Service**<br><br>• Model Submission<br>• Evaluation Pipeline"]24            Services --> Votes["**Vote Service**<br><br>• Vote Management<br>• Data Synchronization"]25            Services --> Board["**Leaderboard Service**<br><br>• Rankings<br>• Performance Metrics"]26        end27 28        Models --> Cache["**Cache Layer**<br><br>• In-Memory Store<br>• Auto Invalidation"]29        Votes --> Cache30        Board --> Cache31 32        Models --> HF["**HuggingFace Hub**<br><br>• Models Repository<br>• Datasets Access"]33        Votes --> HF34        Board --> HF35    end36 37    style Client fill:#f9f,stroke:#333,stroke-width:2px38    style Models fill:#bbf,stroke:#333,stroke-width:2px39    style Votes fill:#bbf,stroke:#333,stroke-width:2px40    style Board fill:#bbf,stroke:#333,stroke-width:2px41    style HF fill:#bfb,stroke:#333,stroke-width:2px42```43 44## 🛠️ HuggingFace Datasets45 46The application uses several datasets on the HuggingFace Hub:47 48### 1. Requests Dataset (`{HF_ORGANIZATION}/requests`)49 50- **Operations**:51  - 📤 `POST /api/models/submit`: Adds a JSON file for each new model submission52  - 📥 `GET /api/models/status`: Reads files to get models status53- **Format**: One JSON file per model with submission details54- **Updates**: On each new model submission55 56### 2. Votes Dataset (`{HF_ORGANIZATION}/votes`)57 58- **Operations**:59  - 📤 `POST /api/votes/{model_id}`: Adds a new vote60  - 📥 `GET /api/votes/model/{provider}/{model}`: Reads model votes61  - 📥 `GET /api/votes/user/{user_id}`: Reads user votes62- **Format**: JSONL with one vote per line63- **Sync**: Bidirectional between local cache and Hub64 65### 3. Contents Dataset (`{HF_ORGANIZATION}/contents`)66 67- **Operations**:68  - 📥 `GET /api/leaderboard`: Reads raw data69  - 📥 `GET /api/leaderboard/formatted`: Reads and formats data70- **Format**: Main dataset containing all scores and metrics71- **Updates**: Automatic after model evaluations72 73### 4. Official Providers Dataset (`{HF_ORGANIZATION}/official-providers`)74 75- **Operations**:76  - 📥 Read-only access for highlighted models77- **Format**: List of models selected by maintainers78- **Updates**: Manual by maintainers79 80## 🛠 Local Development81 82### Prerequisites83 84- Python 3.9+85- [Poetry](https://python-poetry.org/docs/#installation)86 87### Standalone Installation (without Docker)88 89```bash90# Install dependencies91poetry install92 93# Setup configuration94cp .env.example .env95 96# Start development server97poetry run uvicorn app.asgi:app --host 0.0.0.0 --port 7860 --reload98```99 100Server will be available at http://localhost:7860101 102## ⚙️ Configuration103 104| Variable     | Description                          | Default     |105| ------------ | ------------------------------------ | ----------- |106| ENVIRONMENT  | Environment (development/production) | development |107| HF_TOKEN     | HuggingFace authentication token     | -           |108| PORT         | Server port                          | 7860        |109| LOG_LEVEL    | Logging level (INFO/DEBUG/WARNING)   | INFO        |110| CORS_ORIGINS | Allowed CORS origins                 | ["*"]       |111| CACHE_TTL    | Cache Time To Live in seconds        | 300         |112 113## 🔧 Middleware114 115The backend uses several middleware layers for optimal performance and security:116 117- **CORS Middleware**: Handles Cross-Origin Resource Sharing118- **GZIP Middleware**: Compresses responses > 500 bytes119- **Rate Limiting**: Prevents API abuse120- **Caching**: In-memory caching with automatic invalidation121 122## 📝 Logging123 124The application uses a structured logging system with:125 126- Formatted console output127- Different log levels per component128- Request/Response logging129- Performance metrics130- Error tracking131 132## 📁 File Structure133 134```135backend/136├── app/                  # Source code137│   ├── api/             # Routes and endpoints138│   │   └── endpoints/   # Endpoint handlers139│   ├── core/           # Configurations140│   ├── services/       # Business logic141│   └── utils/          # Utilities142└── tests/              # Tests143```144 145## 📚 API146 147Swagger documentation available at http://localhost:7860/docs148 149### Main Endpoints & Data Structures150 151#### Leaderboard152 153- `GET /api/leaderboard/formatted` - Formatted data with computed fields and metadata154 155  ```typescript156  Response {157    models: [{158      id: string,  // eval_name159      model: {160        name: string,  // fullname161        sha: string,  // Model sha162        precision: string,  // e.g. "fp16", "int8"163        type: string,  // e.g. "fined-tuned-on-domain-specific-dataset"164        weight_type: string,165        architecture: string,166        average_score: number,167        has_chat_template: boolean168      },169      evaluations: {170        ifeval: {171          name: "IFEval",172          value: number,  // Raw score173          normalized_score: number174        },175        bbh: {176          name: "BBH",177          value: number,178          normalized_score: number179        },180        math: {181          name: "MATH Level 5",182          value: number,183          normalized_score: number184        },185        gpqa: {186          name: "GPQA",187          value: number,188          normalized_score: number189        },190        musr: {191          name: "MUSR",192          value: number,193          normalized_score: number194        },195        mmlu_pro: {196          name: "MMLU-PRO",197          value: number,198          normalized_score: number199        }200      },201      features: {202        is_not_available_on_hub: boolean,203        is_merged: boolean,204        is_moe: boolean,205        is_flagged: boolean,206        is_official_provider: boolean207      },208      metadata: {209        upload_date: string,210        submission_date: string,211        generation: string,212        base_model: string,213        hub_license: string,214        hub_hearts: number,215        params_billions: number,216        co2_cost: number  // CO₂ cost in kg217      }218    }]219  }220  ```221 222- `GET /api/leaderboard` - Raw data from the HuggingFace dataset223  ```typescript224  Response {225    models: [{226      eval_name: string,227      Precision: string,228      Type: string,229      "Weight type": string,230      Architecture: string,231      Model: string,232      fullname: string,233      "Model sha": string,234      "Average ⬆️": number,235      "Hub License": string,236      "Hub ❤️": number,237      "#Params (B)": number,238      "Available on the hub": boolean,239      Merged: boolean,240      MoE: boolean,241      Flagged: boolean,242      "Chat Template": boolean,243      "CO₂ cost (kg)": number,244      "IFEval Raw": number,245      IFEval: number,246      "BBH Raw": number,247      BBH: number,248      "MATH Lvl 5 Raw": number,249      "MATH Lvl 5": number,250      "GPQA Raw": number,251      GPQA: number,252      "MUSR Raw": number,253      MUSR: number,254      "MMLU-PRO Raw": number,255      "MMLU-PRO": number,256      "Maintainer's Highlight": boolean,257      "Upload To Hub Date": string,258      "Submission Date": string,259      Generation: string,260      "Base Model": string261    }]262  }263  ```264 265#### Models266 267- `GET /api/models/status` - Get all models grouped by status268  ```typescript269  Response {270    pending: [{271      name: string,272      submitter: string,273      revision: string,274      wait_time: string,275      submission_time: string,276      status: "PENDING" | "EVALUATING" | "FINISHED",277      precision: string278    }],279    evaluating: Array<Model>,280    finished: Array<Model>281  }282  ```283- `GET /api/models/pending` - Get pending models only284- `POST /api/models/submit` - Submit model285 286  ```typescript287  Request {288    user_id: string,289    model_id: string,290    base_model?: string,291    precision?: string,292    model_type: string293  }294 295  Response {296    status: string,297    message: string298  }299  ```300 301- `GET /api/models/{model_id}/status` - Get model status302 303#### Votes304 305- `POST /api/votes/{model_id}` - Vote306 307  ```typescript308  Request {309    vote_type: "up" | "down",310    user_id: string  // HuggingFace username311  }312 313  Response {314    success: boolean,315    message: string316  }317  ```318 319- `GET /api/votes/model/{provider}/{model}` - Get model votes320  ```typescript321  Response {322    total_votes: number,323    up_votes: number,324    down_votes: number325  }326  ```327- `GET /api/votes/user/{user_id}` - Get user votes328  ```typescript329  Response Array<{330    model_id: string,331    vote_type: string,332    timestamp: string333  }>334  ```335 336## 🔒 Authentication337 338The backend uses HuggingFace token-based authentication for secure API access. Make sure to:339 3401. Set your HF_TOKEN in the .env file3412. Include the token in API requests via Bearer authentication3423. Keep your token secure and never commit it to version control343 344## 🚀 Performance345 346The backend implements several optimizations:347 348- In-memory caching with configurable TTL (Time To Live)349- Batch processing for model evaluations350- Rate limiting for API endpoints351- Efficient database queries with proper indexing352- Automatic cache invalidation for votes353