Datasetpublic

berryccc1/EVM-QuestBench

EVM-QuestBench EVM-QuestBench is an execution-grounded benchmark for evaluating whether large language models and AI agents can translate natural-language blockchain intent into executable transaction code that produces the intended EVM state transition. Released with the ACL 2026 Long Paper by Pei Yang, Wanyi Chen, Ke Wang, Lynn Ai, Eric Yang, and Tianyu Shi, the benchmark contains 107 expert-authored tasks: 62 atomic tasks and 45 composite workflows. Unlike code-similarity… See the full description on the dataset page: https://huggingface.co/datasets/berryccc1/EVM-QuestBench.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes29downloads

rootCITATION.cff

CITATION.cff69 linesDownload Raw Back to root

1cff-version: 1.2.02title: "EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation"3message: "If you use EVM-QuestBench, please cite the ACL 2026 Long Paper."4type: software5authors:6  - given-names: Pei7    family-names: Yang8    affiliation: Gradient9    equal-contribution: true10  - given-names: Wanyi11    family-names: Chen12    affiliation: Soochow University13    equal-contribution: true14  - given-names: Ke15    family-names: Wang16    affiliation: Gradient17  - given-names: Lynn18    family-names: Ai19    affiliation: Gradient20  - given-names: Eric21    family-names: Yang22    affiliation: Gradient23  - given-names: Tianyu24    family-names: Shi25    affiliation: Gradient26    corresponding: true27repository-code: "https://github.com/OpenEdgeHQ/EVM-quest-bench"28url: "https://openedgehq.github.io/EVM-quest-bench/"29repository-artifact: "https://github.com/OpenEdgeHQ/EVM-quest-bench"30license: MIT31version: "1.0.0"32date-released: 2026-07-0233identifiers:34  - type: doi35    value: "10.18653/v1/2026.acl-long.1642"36    description: "ACL Anthology publication DOI"37  - type: arxiv38    value: "2601.06565"39keywords:40  - EVM41  - blockchain42  - large language models43  - transaction code generation44  - benchmark45  - agent evaluation46  - ACL 202647preferred-citation:48  type: article49  title: "EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation"50  authors:51    - given-names: Pei52      family-names: Yang53    - given-names: Wanyi54      family-names: Chen55    - given-names: Ke56      family-names: Wang57    - given-names: Lynn58      family-names: Ai59    - given-names: Eric60      family-names: Yang61    - given-names: Tianyu62      family-names: Shi63  journal: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics"64  year: 202665  volume: 166  start: 3551367  end: 3552968  doi: "10.18653/v1/2026.acl-long.1642"69