ByteDance-Seed/EdgeBench

<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/datasets/ByteDance-Seed/EdgeBench/resolve/main/assets/logo-dark.png"> <img src="assets/logo.jpg" alt="ByteDance Seed" width="420"> </picture> </p>

<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/datasets/ByteDance-Seed/EdgeBench/resolve/main/assets/title-dark.svg"> <img src="assets/title.svg" alt="EdgeBench" width="280"> </picture> </p>

<br>

<!-- <p align="center"> <strong>Measuring How AI Agents Learn from Real-World Environments</strong> </p>

<p align="center"> 134 real-world tasks &nbsp;|&nbsp; 51 open-source &nbsp;|&nbsp; 6 capability categories &nbsp;|&nbsp; 38,000+ hours of agent interaction </p> -->

<p align="center"> <a href="https://edge-bench.org/"><img src="https://img.shields.io/badge/Project-edge--bench.org-blue" alt="Project"></a> <a href="https://arxiv.org/abs/2607.05155"><img src="https://img.shields.io/badge/Tech%20Report-PDF-red?logo=adobeacrobatreader" alt="Tech Report"></a> <a href="https://github.com/ByteDance-Seed/EdgeBench"><img src="https://img.shields.io/badge/GitHub-EdgeBench-green?logo=github" alt="GitHub"></a> <a href="https://bytedance-seed.github.io/EdgeBench/"><img src="https://img.shields.io/badge/Docs-SForge%20Harness-purple" alt="Docs"></a> <a href="https://github.com/ByteDance-Seed/EdgeBench/blob/main/assets/wechat_qr.jpg"><img src="https://img.shields.io/badge/WeChat-Group-07C160?logo=wechat&logoColor=white" alt="WeChat Group"></a> <a href="https://discord.gg/p2JZ26ku8"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white" alt="Discord"></a> </p>

Overview

EdgeBench is a benchmark of 134 real-world tasks for evaluating how autonomous AI agents learn from real-world environments. Instead of measuring one-shot performance, EdgeBench places agents in executable task environments with realistic, multi-level feedback and lets them iterate for 12+ hours per task — tracking the full trajectory of improvement, not just the final score. We publicly release 51 tasks along with the full evaluation framework.

Analyzing ~38,000 hours of agent interaction on all 134 tasks, we find that performance follows a log-sigmoid scaling law as a function of interaction time ($R^2 = 0.998$). See the tech report for details.

<p align="center"> <img src="assets/figfull136curvefitsideby_side.png" alt="Log-sigmoid scaling fit across 134 tasks" width="800"> </p>

Leaderboard

Full Benchmark (134 tasks)

<details> <summary><b>Category Scores @12h (134 tasks)</b></summary>

</details>

Open-Source Subset (51 tasks)

<details> <summary><b>Category Scores @12h (51 tasks)</b></summary>

</details>

<details> <summary><b>Per-Task Scores by Time Budget (51 tasks)</b></summary>

Each model cell reports scores at @2h / @4h / @6h / @8h / @10h / @12h. Missing valid results are shown as —.

</details>

Task Taxonomy

EdgeBench contains 134 realistic, diverse tasks spanning six capability categories, of which 51 are publicly released. Each task is designed as a day-scale challenge with a performance ceiling high enough that no current agent can saturate it. Recorded human expert effort averages 57.2 hours per task (up to 320 hours).

<p align="center"> <img src="assets/edgebench_taxonomy.png" alt="EdgeBench Task Taxonomy" width="850"> </p>

Evaluation Harness: SForge

EdgeBench is powered by **SForge**, a two-container evaluation harness built for long-horizon agent evaluation. See the SForge documentation for setup and usage instructions.

Citation

If you find EdgeBench useful in your research, please cite our tech report:

@misc{edgebench2026,
  title  = {EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments},
  author = {Deyao Zhu and Xin Zhou and Shengling Qin and Xuekai Zhu and Hangliang Ding and Shu Zhong and others},
  year   = {2026},
  url    = {https://arxiv.org/abs/2607.05155},
}

License

EdgeBench task datasets are released under CC BY 4.0.

Contact

To evaluate on the full 134-task suite, please contact zhongshu@bytedance.com.

<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/datasets/ByteDance-Seed/EdgeBench/resolve/main/assets/logo-dark.png"> <img src="assets/logo.jpg" alt="ByteDance Seed" width="200"> </picture> <br> _{Built by <a href="https://github.com/ByteDance-Seed">ByteDance Seed</a>} </p>