microsoft/BizGenEval-Leaderboard

BizGenEval Leaderboard

This repository hosts the Hugging Face leaderboard for BizGenEval, the benchmark introduced in *BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation*.

Primary project resources:

• —Project page: https://aka.ms/BizGenEval
• —GitHub: https://github.com/microsoft/BizGenEval
• —Dataset: https://huggingface.co/datasets/microsoft/BizGenEval

The codebase supports:

1. 1.LOCAL_DEV mode (no HF permission required): reads/writes local namespaced paths under eval-queue/ and eval-results/.
2. 2.HF mode (with permission): syncs datasets from the Hub and uploads queue requests.

1) Local development quick start (no HF permission)

Step 1. Create and activate virtualenv

cd /Users/clarencestark/code/BizGenEval-Leaderboard
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Step 2. Bootstrap local demo data

python3 scripts/bootstrap_local_dev.py

This will create:

• —eval-queue/bizgeneval/requests/microsoft/Phi-4o-mini_eval_request_False_float16_Original.json
• —eval-results/bizgeneval/results/microsoft/Phi-4o-mini/summary.json

Step 3. Launch in local mode

export LOCAL_DEV=1
python3 app.py

In LOCAL_DEV mode:

• —snapshot_download is skipped.
• —Model-card/tokenizer checks are skipped during submission.
• —New submissions are written to local eval-queue/bizgeneval/requests/ only (no upload).

2) Result file format supported

The leaderboard parser currently supports two formats:

A) BizGenEval summary format (recommended)

Put a summary.json under:

eval-results/bizgeneval/results/<org>/<model>/summary.json

Example:

{
  "model_name": "microsoft/Phi-4o-mini",
  "model_sha": "main",
  "by_domain": {
    "slides": {"error_score": 0.8125},
    "webpage": {"error_score": 0.845},
    "poster": {"error_score": 0.7875},
    "chart": {"error_score": 0.8025},
    "scientific_figure": {"error_score": 0.77}
  },
  "by_dimension": {
    "layout": {"error_score": 0.835},
    "attribute": {"error_score": 0.805},
    "text": {"error_score": 0.79},
    "knowledge": {"error_score": 0.775}
  }
}

error_score can be either 0~1 or 0~100; both are accepted and normalized to a displayed 0~100 scale.

B) Legacy template format

Legacy config/results JSON is still accepted for compatibility.

3) Queue file format

Queue entries are JSON files in:

eval-queue/bizgeneval/requests/<org>/*.json

A typical file contains:

• —model
• —revision
• —precision
• —weight_type
• —status (PENDING, RUNNING, FINISHED*)
• —metadata (license, params, likes, ...)

4) Config knobs

Main config file: src/envs.py

• —LOCAL_DEV (env): 1/true/on to enable local mode
• —HF_OWNER (env, optional): owner fallback
• —PROJECT_NAMESPACE (env, optional): defaults to bizgeneval
• —HF_SPACE_REPO (env, optional)
• —HF_QUEUE_REPO (env, optional)
• —HF_RESULTS_REPO (env, optional)
• —HF_TOKEN (env): required only for Hub sync/upload

Default repo names are:

• —Space: microsoft/BizGenEval-Leaderboard
• —Queue dataset: demo-leaderboard-backend/requests
• —Results dataset: demo-leaderboard-backend/results

5) Key code locations

• —Columns and UI display fields: src/display/utils.py
• —Result parser: src/leaderboard/read_evals.py
• —DataFrame build logic: src/populate.py
• —Submission validation/upload behavior: src/submission/submit.py
• —Task definitions and page text: src/about.py