ByteDance-Seed/EdgeBench
Overview EdgeBench is a benchmark of 134 real-world tasks for evaluating how autonomous AI agents learn from real-world environments. Instead of measuring one-shot performance, EdgeBench places agents in executable task environments with realistic, multi-level feedback and lets them iterate for 12+ hours per task — tracking the full trajectory of improvement, not just the final score. We publicly release 51 tasks… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/EdgeBench.
847.5k
1{2 "task_id": "apple_incremental_game",3 "name": "Apple Incremental Game",4 "category": "Combinatorial Optimization",5 "base_image": "python310",6 "platform": "linux/amd64",7 "internet": false,8 "cwd": "/home/workspace/apple_incremental_game",9 "submit_paths": [10 "solution.py"11 ],12 "work": {13 "image_tag": "d3c6ed381c59",14 "specs_dir": "/home/workspace/apple_incremental_game",15 "agent_query": "## Apple Incremental Game - Machine Production (AHC058)\n\nWrite `solution.py` in the project root that reads from stdin and writes to stdout.\n\n---\n\n## Problem Overview\n\nRead `README.md` and `tools/README.md` for full problem details. A baseline `solution.py` already exists (it produces syntactically valid but low-quality output). Your job is to improve it.\n\n---\n\n## Evaluation\n\nYour solution is scored on **50 fixed test cases**. Final score = sum of individual case scores. **Higher is better.**\n\n---\n\n## Local Testing\n\nGenerate local random tests with `./tools/bin/gen <seed>`, using seeds in the range **0..10000** only.\n\n```bash\n# Generate a random test case (seed-based, deterministic)\n./tools/bin/gen 0 > input.txt\n\n# Run your solution\npython3 solution.py < input.txt > output.txt\n\n# Score output (Higher is better)\n./tools/bin/tester input.txt output.txt\n# Outputs to stderr: Score = <N>\n```\n\n---\n\n## Rules\n\n- Write your solution as `solution.py` in the project root directory\n- Do NOT modify files in `tools/`\n- Use `tools/bin/gen` and `tools/bin/tester` for local testing\n- For local scoring, use only `./tools/bin/tester`; do not use `tools/src/verifier.py` for scores\n- Your program should read from stdin and write to stdout\n- Run your solution to completion and verify with the tester before finishing\n"16 },17 "judge": {18 "image_tag": "16968d8ec7f2",19 "eval_cmd": "cd /home/workspace/apple_incremental_game && python3 /tmp/eval_apple_incremental_game.py",20 "eval_timeout": 600,21 "parser": "score_sum",22 "score_direction": "maximize",23 "selection": "score_first",24 "rescale": {25 "kind": "piecewise_max",26 "baseline": 45000000.0,27 "rank30": 262328915.6667,28 "rank1": 282791094.0,29 "super_anchor": 289952856.416730 }31 }32}33