dari-ai/vite-selfbench

Vite Selfbench

Vite Selfbench is a 27-task software-engineering benchmark for coding agents, packaged for the Harbor evaluation framework. Each task asks an agent to implement a change in a frozen revision of `vitejs/vite`, then checks the resulting patch with task-specific tests. Harbor provides isolated task environments and runs verification separately from the agent.

This repository contains the raw evaluation only. It does not include model outputs, scores, costs, or benchmark result artifacts.

Use

Download the raw task package with the Hugging Face CLI:

hf download dari-ai/vite-selfbench --repo-type dataset --local-dir vite-selfbench

After installing and configuring Harbor and the coding-agent credentials you want to use, run the full collection:

harbor run --path vite-selfbench --agent pi

To run one task instead, point --path at that task directory:

harbor run \
  --path vite-selfbench/vite-22642-input-option \
  --agent pi

The dataset intentionally does not include agent credentials or agent configuration. See the Harbor documentation for installation, supported agents, and configuration options.

Contents

Each top-level task directory contains:

• —task.toml: the task configuration and source metadata.
• —instruction.md: the request shown to the coding agent.
• —environment/: the frozen Vite repository and build environment.
• —tests/: the tests used by the verifier.
• —solution/: the reference patch and solution script.
• —.selfbench-manifest.json: task provenance and integrity checksums.

tasks.tsv provides a compact manifest of all 27 task IDs, difficulty labels, and definition/test/solution checksums.

Reproducibility

Each task pins a Vite base commit and includes its repository archive, so it does not depend on future upstream changes. The task definitions use Harbor schema version 1.4 and were generated with Selfbench compiler revision 23. These version fields record how the tasks were packaged; they are not additional installation requirements.

Important evaluation note

The repository includes verifier tests and reference solutions because this is the complete raw evaluation package. Do not expose tests/ or solution/ to an agent: they are evaluation assets, not part of the coding-agent workspace. Harbor keeps verification separate when the task runs normally; preserve that separation if adapting these tasks to another framework.

Licensing

The embedded Vite source is distributed under its upstream license. Other files retain the terms of their respective sources. Review the included upstream license and source metadata before redistributing modified versions.