REAL-Lab-Imperial/eval-unlearn

Eval-Unlearn Leaderboard

Interactive leaderboard for eval-unlearn, a benchmarking framework for concept unlearning in text-to-image diffusion models.

What this Space does

• —Leaderboard tab: a maintainer-curated reference table of the built-in unlearning techniques, plus a community submissions table populated by the Submit tab.
• —Submit tab: paste in any public HF diffusers model repo and a target concept, and this Space runs a cheap online evaluation on ZeroGPU: CLIP Score, TIFA, FID, ERR (nudity only), UA/IRA (nudity/violence only), and ASR-I2P (when the concept is one of I2P's 7 categories). Each metric uses a small default number of images/prompts, adjustable per metric in the "Advanced: dataset size per metric" section (with a "use defaults" option that leaves the current defaults untouched). Generated images are never stored or published, only the aggregate scores.
• —Adversarial ASR cannot be computed on this Space. Ring-A-Bell, MMA-Diffusion and P4D need far more compute than a single ZeroGPU call here can give. Run them yourself, locally or on your own hardware, with the eval-unlearn package (pip install eval-unlearn, see PyPI), for example eval-unlearn run --config ... --hf-repo ..., then attach the resulting *_report_full.json on the Submit tab. It is schema-validated and merged in as self-reported, offline-computed columns.

There is no manual approval queue before a submission runs (a deliberate v1 choice). The guardrails instead are requiring HF sign-in to submit, a cheap preflight check that the repo is actually a diffusers pipeline, and a per-account daily submission cap. See src/evaluation/validate.py.

Code layout

• —app.py: the two-tab Gradio UI.
• —src/evaluation/runner.py: the ZeroGPU-decorated per-metric driver, built directly on eval_unlearn.registry.get_technique/get_metric (bypassing eval-unlearn's runners so each metric gets its own short GPU allocation and no images ever touch disk). Also defines DEFAULT_SIZES/MAX_SIZES, the per-metric dataset size defaults and slider ceilings used by the Submit tab.
• —src/evaluation/validate.py: preflight repo checks, offline-report validation, rate limiting.
• —src/leaderboard/results_store.py: reads/writes submission rows against the `REAL-Lab-Imperial/eval-unlearn-results` dataset (one JSON file per submission).
• —data/ua_ira/: the curated target/retain prompt CSVs UA/IRA needs (bundled from eval-unlearn's examples/data/).

Requirements to run

• —ZeroGPU hardware enabled on this Space (Settings, then Hardware).
• —An HF_TOKEN secret with write access to the eval-unlearn-results dataset repo.