LogicStar/BaxBench

1---2license: mit3configs:4- config_name: default5  data_files:6  - split: test7    path: data/test-*8dataset_info:9  features:10  - name: task_id11    dtype: string12  - name: scenario_id13    dtype: string14  - name: env_id15    dtype: string16  - name: api_specification17    dtype: string18  - name: text_specification19    dtype: string20  - name: short_app_description21    dtype: string22  - name: scenario_instructions23    dtype: string24  - name: needs_db25    dtype: bool26  - name: needs_secret27    dtype: bool28  - name: needed_packages29    struct:30    - name: JavaScript31      sequence: string32    - name: _all_33      sequence: string34  - name: potential_cwes35    sequence: int6436  - name: env_language37    dtype: string38  - name: env_extension39    dtype: string40  - name: env_framework41    dtype: string42  - name: env_multifile43    dtype: bool44  - name: code_filename45    dtype: string46  - name: entrypoint_cmd47    dtype: string48  - name: allowed_packages49    dtype: string50  - name: env_instructions51    dtype: string52  - name: port53    dtype: int6454  splits:55  - name: test56    num_bytes: 183026257    num_examples: 39258  download_size: 7054059  dataset_size: 183026260task_categories:61- text-generation62tags:63- code64- security65- benchmark66size_categories:67- n<1K68---69 70### Dataset Summary71BaxBench is a coding benchmark constructed to measure the ability of code generation models and agents to generate correct and secure code. It consists of 392 backend development tasks, which are constructed by combining 28 scenarios that describe the backend functionalities to implement and 14 backend frameworks defining the implementation tools. To assess the correctness and security of the solutions, the benchmark uses end-to-end functional tests and practical securtiy exploits.72 73The dataset is released as part of the paper and benchmark: [BaxBench: Can LLMs generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844).74 75The dataset contains all necessary artifacts to reproduce the evaluation prompts used in our paper. Further, it enables the testing of different prompt structures or models by forming new prompt types, e.g., for testing code agents.76 77For details on reproducing our results, or testing your models on the same prompts, please refer to our [paper](https://arxiv.org/abs/2502.11844) or [code repository](https://github.com/logic-star-ai/baxbench).78 79To test your generated solutions, please follow the instructions in our [code repository](https://github.com/logic-star-ai/baxbench).80 81For more details on the construction of BaxBench, large-scale model evaluation results, and detailed analyses, please see our [paper](https://arxiv.org/abs/2502.11844) or visit our [website](https://baxbench.com).82 83### Citation84 85**BibTeX:**86```87@article{vero2025baxbenchllmsgeneratecorrect,88    title={BaxBench: Can LLMs Generate Correct and Secure Backends?},89    author={Mark Vero and Niels Mündler and Victor Chibotaru and Veselin Raychev and Maximilian Baader and Nikola Jovanović and Jingxuan He and Martin Vechev},90    year={2025},91    eprint={2502.11844},92    archivePrefix={arXiv},93}94```