whitphx/transformersjs-bench-leaderboard

Transformers.js Benchmark Leaderboard

A Gradio-based leaderboard that displays benchmark results from a HuggingFace Dataset repository.

Features

• —📊 Display benchmark results in a searchable/filterable table
• —🔍 Filter by model name, task, platform, device, mode, and dtype
• —🔄 Refresh data on demand from HuggingFace Dataset
• —📈 View performance metrics (load time, inference time, p50/p90 percentiles)

Setup

1. 1.Install dependencies:

   uv sync

1. 1.Configure environment variables:

   cp .env.example .env

Edit .env and set:

• —HF_DATASET_REPO: Your HuggingFace dataset repository (e.g., username/transformersjs-benchmarks)
• —HF_TOKEN: Your HuggingFace API token (optional, for private datasets)

Usage

Run the leaderboard:

uv run python -m leaderboard.app

Or using the installed script:

uv run leaderboard

The leaderboard will be available at: http://localhost:7861

Data Format

The leaderboard reads JSONL files from the HuggingFace Dataset repository. Each line should be a JSON object with the following structure:

{
  "id": "benchmark-id",
  "platform": "web",
  "modelId": "Xenova/all-MiniLM-L6-v2",
  "task": "feature-extraction",
  "mode": "warm",
  "repeats": 3,
  "batchSize": 1,
  "device": "wasm",
  "browser": "chromium",
  "dtype": "fp32",
  "headed": false,
  "status": "completed",
  "timestamp": 1234567890,
  "result": {
    "metrics": {
      "load_ms": {"p50": 100, "p90": 120},
      "first_infer_ms": {"p50": 10, "p90": 15},
      "subsequent_infer_ms": {"p50": 8, "p90": 12}
    },
    "environment": {
      "cpuCores": 10,
      "memory": {"deviceMemory": 8}
    }
  }
}

Deployment on Hugging Face Spaces

This leaderboard is designed to be deployed on Hugging Face Spaces using the Gradio SDK.

Quick Deploy

1. 1.Create a new Space on Hugging Face:
2. 2.Go to https://huggingface.co/new-space
3. 3.Choose Gradio as the SDK
4. 4.Set the Space name (e.g., transformersjs-benchmark-leaderboard)

1. 1.Upload files to your Space:

   # Clone your Space repository
   git clone https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
   cd YOUR_SPACE_NAME

   # Copy leaderboard files
   cp -r /path/to/leaderboard/* .

   # Commit and push
   git add .
   git commit -m "Initial leaderboard deployment"
   git push

1. 1.Configure Space secrets:
2. 2.Go to your Space settings → Variables and secrets
3. 3.Add the following secrets:
4. 4.HF_DATASET_REPO: Your dataset repository (e.g., username/benchmark-results)
5. 5.HF_TOKEN: Your HuggingFace API token (for private datasets)

1. 1.Space will automatically deploy and be available at:

   https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME

Space Configuration

The Space is configured via the YAML frontmatter in README.md:

---
title: Transformers.js Benchmark Leaderboard
emoji: 🏆
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: src/leaderboard/app.py
pinned: false
---

Key configuration options:

• —sdk: Must be gradio for Gradio apps
• —sdk_version: Gradio version (matches your pyproject.toml)
• —app_file: Path to the main Python file (relative to repository root)
• —pinned: Set to true to pin the Space on your profile

Requirements

The Space will automatically install dependencies from pyproject.toml:

• —gradio>=5.9.1
• —pandas
• —huggingface-hub
• —python-dotenv

Environment Variables

Set these in your Space settings or in a .env file (not recommended for production):

Auto-Restart

Spaces automatically restart when:

• —Code is pushed to the repository
• —Dependencies are updated
• —Environment variables are changed

Monitoring

• —View logs in the Space's Logs tab
• —Check status in the Settings tab
• —Monitor resource usage (CPU, memory)

Development

The leaderboard is built with:

• —Gradio: Web UI framework
• —Pandas: Data manipulation
• —HuggingFace Hub: Dataset loading

Local Development

1. 1.Install dependencies:

   uv sync

1. 1.Set environment variables:

   export HF_DATASET_REPO="your-username/benchmark-results"
   export HF_TOKEN="your-hf-token"  # Optional

1. 1.Run locally:

   uv run python -m leaderboard.app

1. 1.Access at: http://localhost:7861