Apppublic

HF-slyfox/harness

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes

App README

slyfox harness

A SWE-bench-style benchmark for measuring whether the slyfox Claude Code plugin's injected prior-maintainer trace excerpts actually help the model solve real tasks.

What's here

100 tasks, each derived from a real merged perf-flavored PR in huggingface/transformers. For each task:

—issue_title / issue_body — a plausible bug report a user might have filed before the fix existed (LLM-polished, anti-leak: no internal function/class/file names from the PR)
—base_commit — git checkout this to recreate the "before" state
—target_files — files the gold patch touches
—diff path — the merged PR's own diff (the answer)
—score / score_breakdown — hardness ranking (size, perf signal, modeling-code overlap)

Browse

The Gradio UI lists all 100 tasks. Click into one to see the synthesised issue and the first lines of the gold patch.

Schema

See benchmark.jsonl for the full per-entry schema. Key fields:

field	meaning
`task_id`	`transformers-<pr#>`
`repo`	`huggingface/transformers`
`pr_number`, `pr_url`, `pr_title`	provenance
`base_commit`	parent of the merge commit
`target_files`	files the gold patch touches
`diff_path`	relative path to gold patch under `diffs/`
`issue_title`, `issue_body`	the synthesised fake issue
`pr_created_at`, `pr_closed_at`	for time-based leak prevention

Leak prevention

When running a task against the slyfox traces dataset, time-filter the traces to sessions where `started_at < pr_created_at` for that task — otherwise the model could see the actual session that fixed the bug.

Reproduce / extend

The mining and polishing scripts live alongside the slyfox plugin source at https://github.com/huggingface/slyfox/tree/main/harness — bump --target on build_benchmark.py to mine more, then polish_issues.py polishes them.