bigcode/humanevalpack

1---2license: mit3pretty_name: HumanEvalPack4language_creators:5- expert-generated6multilinguality:7- multilingual8language:9- code10tags:11- code12dataset_info:13- config_name: cpp14  features:15  - name: task_id16    dtype: string17  - name: prompt18    dtype: string19  - name: declaration20    dtype: string21  - name: canonical_solution22    dtype: string23  - name: buggy_solution24    dtype: string25  - name: bug_type26    dtype: string27  - name: failure_symptoms28    dtype: string29  - name: entry_point30    dtype: string31  - name: import32    dtype: string33  - name: test_setup34    dtype: string35  - name: test36    dtype: string37  - name: example_test38    dtype: string39  - name: signature40    dtype: string41  - name: docstring42    dtype: string43  - name: instruction44    dtype: string45  splits:46  - name: test47    num_bytes: 46911148    num_examples: 16449  download_size: 19398150  dataset_size: 46911151- config_name: go52  features:53  - name: task_id54    dtype: string55  - name: prompt56    dtype: string57  - name: declaration58    dtype: string59  - name: canonical_solution60    dtype: string61  - name: buggy_solution62    dtype: string63  - name: bug_type64    dtype: string65  - name: failure_symptoms66    dtype: string67  - name: entry_point68    dtype: string69  - name: import70    dtype: string71  - name: test_setup72    dtype: string73  - name: test74    dtype: string75  - name: example_test76    dtype: string77  - name: signature78    dtype: string79  - name: docstring80    dtype: string81  - name: instruction82    dtype: string83  splits:84  - name: test85    num_bytes: 46323486    num_examples: 16487  download_size: 19839488  dataset_size: 46323489- config_name: java90  features:91  - name: task_id92    dtype: string93  - name: prompt94    dtype: string95  - name: declaration96    dtype: string97  - name: canonical_solution98    dtype: string99  - name: buggy_solution100    dtype: string101  - name: bug_type102    dtype: string103  - name: failure_symptoms104    dtype: string105  - name: entry_point106    dtype: string107  - name: import108    dtype: string109  - name: test_setup110    dtype: string111  - name: test112    dtype: string113  - name: example_test114    dtype: string115  - name: signature116    dtype: string117  - name: docstring118    dtype: string119  - name: instruction120    dtype: string121  splits:122  - name: test123    num_bytes: 589440124    num_examples: 164125  download_size: 210440126  dataset_size: 589440127- config_name: js128  features:129  - name: task_id130    dtype: string131  - name: prompt132    dtype: string133  - name: declaration134    dtype: string135  - name: canonical_solution136    dtype: string137  - name: buggy_solution138    dtype: string139  - name: bug_type140    dtype: string141  - name: failure_symptoms142    dtype: string143  - name: entry_point144    dtype: string145  - name: import146    dtype: string147  - name: test_setup148    dtype: string149  - name: test150    dtype: string151  - name: example_test152    dtype: string153  - name: signature154    dtype: string155  - name: docstring156    dtype: string157  - name: instruction158    dtype: string159  splits:160  - name: test161    num_bytes: 435189162    num_examples: 164163  download_size: 194044164  dataset_size: 435189165- config_name: python166  features:167  - name: task_id168    dtype: string169  - name: prompt170    dtype: string171  - name: declaration172    dtype: string173  - name: canonical_solution174    dtype: string175  - name: buggy_solution176    dtype: string177  - name: bug_type178    dtype: string179  - name: failure_symptoms180    dtype: string181  - name: entry_point182    dtype: string183  - name: import184    dtype: string185  - name: test_setup186    dtype: string187  - name: test188    dtype: string189  - name: example_test190    dtype: string191  - name: signature192    dtype: string193  - name: docstring194    dtype: string195  - name: instruction196    dtype: string197  splits:198  - name: test199    num_bytes: 423013200    num_examples: 164201  download_size: 191279202  dataset_size: 423013203- config_name: rust204  features:205  - name: task_id206    dtype: string207  - name: prompt208    dtype: string209  - name: declaration210    dtype: string211  - name: canonical_solution212    dtype: string213  - name: buggy_solution214    dtype: string215  - name: bug_type216    dtype: string217  - name: failure_symptoms218    dtype: string219  - name: entry_point220    dtype: string221  - name: import222    dtype: string223  - name: test_setup224    dtype: string225  - name: test226    dtype: string227  - name: example_test228    dtype: string229  - name: signature230    dtype: string231  - name: docstring232    dtype: string233  - name: instruction234    dtype: string235  splits:236  - name: test237    num_bytes: 450539238    num_examples: 164239  download_size: 168464240  dataset_size: 450539241configs:242- config_name: cpp243  data_files:244  - split: test245    path: cpp/test-*246- config_name: go247  data_files:248  - split: test249    path: go/test-*250- config_name: java251  data_files:252  - split: test253    path: java/test-*254- config_name: js255  data_files:256  - split: test257    path: js/test-*258- config_name: python259  data_files:260  - split: test261    path: python/test-*262  default: true263- config_name: rust264  data_files:265  - split: test266    path: rust/test-*267---268 269![Octopack](https://github.com/bigcode-project/octopack/blob/31f3320f098703c7910e43492c39366eeea68d83/banner.png?raw=true)270 271# Dataset Card for HumanEvalPack272 273## Table of Contents274- [Table of Contents](#table-of-contents)275- [Dataset Description](#dataset-description)276  - [Dataset Summary](#dataset-summary)277  - [Languages](#languages)278- [Dataset Structure](#dataset-structure)279  - [Data Instances](#data-instances)280  - [Data Fields](#data-fields)281  - [Data Splits](#data-splits)282- [Dataset Creation](#dataset-creation)283  - [Curation Rationale](#curation-rationale)284  - [Source Data](#source-data)285  - [Annotations](#annotations)286- [Additional Information](#additional-information)287  - [Licensing Information](#licensing-information)288  - [Citation Information](#citation-information)289  - [Contributions](#contributions)290 291## Dataset Description292 293- **Repository:** https://github.com/bigcode-project/octopack294- **Paper:** [OctoPack: Instruction Tuning Code Large Language Models](https://arxiv.org/abs/2308.07124)295- **Point of Contact:** [Niklas Muennighoff](mailto:n.muennighoff@gmail.com)296 297### Dataset Summary298 299> HumanEvalPack is an extension of OpenAI's HumanEval to cover 6 total languages across 3 tasks. The Python split is exactly the same as OpenAI's Python HumanEval. The other splits are translated by humans (similar to HumanEval-X but with additional cleaning, see [here](https://github.com/bigcode-project/octopack/tree/main/evaluation/create/humaneval-x#modifications-muennighoff)). Refer to the [OctoPack paper](https://arxiv.org/abs/2308.07124) for more details.300> 301- **Languages:** Python, JavaScript, Java, Go, C++, Rust302- **OctoPack🐙🎒:**303 304<table>305<tr>306<th>Data</t> 307<td><a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack</a></td>308<td>4TB of GitHub commits across 350 programming languages</td>309</tr>310<tr>311<th></t> 312<td><a href=https://huggingface.co/datasets/bigcode/commitpackft>CommitPackFT</a></td>313<td>Filtered version of CommitPack for high-quality commit messages that resemble instructions</td>314</tr>315<tr>316<th>Model</t> 317<td><a href=https://huggingface.co/bigcode/octocoder>OctoCoder</a></td>318<td>StarCoder (16B parameters) instruction tuned on CommitPackFT + OASST</td>319</tr>320<tr>321<th></t> 322<td><a href=https://huggingface.co/bigcode/octogeex>OctoGeeX</a></td>323<td>CodeGeeX2 (6B parameters) instruction tuned on CommitPackFT + OASST</td>324</tr>325<tr>326<th>Evaluation</t> 327<td><a href=https://huggingface.co/datasets/bigcode/humanevalpack>HumanEvalPack</a></td>328<td>Extension of OpenAI's HumanEval to cover 3 scenarios across 6 languages</td>329</tr>330</table>331 332## Usage333 334```python335# pip install -q datasets336from datasets import load_dataset337# Languages: "python", "js", "java", "go", "cpp", "rust"338ds = load_dataset("bigcode/humanevalpack", "python")["test"]339ds[0]340```341 342## Dataset Structure343 344 345### Data Instances346 347 348An example looks as follows:349 350```json351{352  "task_id": "Python/0",353  "prompt": "from typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    \"\"\" Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n    \"\"\"\n",354  "declaration": "from typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n",355  "canonical_solution": "    for idx, elem in enumerate(numbers):\n        for idx2, elem2 in enumerate(numbers):\n            if idx != idx2:\n                distance = abs(elem - elem2)\n                if distance < threshold:\n                    return True\n\n    return False\n",356  "buggy_solution": "    for idx, elem in enumerate(numbers):\n        for idx2, elem2 in enumerate(numbers):\n            if idx != idx2:\n                distance = elem - elem2\n                if distance < threshold:\n                    return True\n\n    return False\n",357  "bug_type": "missing logic",358  "failure_symptoms": "incorrect output",359  "entry_point": "has_close_elements",360  "import": ""361  "test_setup": ""362  "test": "\n\n\n\n\ndef check(has_close_elements):\n    assert has_close_elements([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\n    assert has_close_elements([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False\n    assert has_close_elements([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True\n    assert has_close_elements([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False\n    assert has_close_elements([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True\n    assert has_close_elements([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True\n    assert has_close_elements([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False\n\ncheck(has_close_elements)",363  "example_test": "def check(has_close_elements):\n    assert has_close_elements([1.0, 2.0, 3.0], 0.5) == False\n    assert has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) == True\ncheck(has_close_elements)\n",364  "signature": "has_close_elements(numbers: List[float], threshold: float) -> bool",365  "docstring": "Check if in given list of numbers, are any two numbers closer to each other than\ngiven threshold.\n>>> has_close_elements([1.0, 2.0, 3.0], 0.5)\nFalse\n>>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\nTrue",366  "instruction": "Write a Python function `has_close_elements(numbers: List[float], threshold: float) -> bool` to solve the following problem:\nCheck if in given list of numbers, are any two numbers closer to each other than\ngiven threshold.\n>>> has_close_elements([1.0, 2.0, 3.0], 0.5)\nFalse\n>>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\nTrue"367}368```369 370### Data Fields371 372The data fields are the same among all splits:373- `task_id`: Indicates the language (Python/JavaScript/Java/Go/C++/Rust) and task id (from 0 to 163) of the problem374- `prompt`: the prompt for models relying on code continuation375- `declaration`: the declaration of the function (same as prompt but without the docstring)376- `canonical_solution`: the correct solution passing all unit tests for the problem377- `buggy_solution`: same as `canonical_solution` but with a subtle human-written bug causing the unit tests to fail378- `bug_type`: the type of the bug in `buggy_solution` (one of [`missing logic`, `excess logic`, `value misuse`, `operator misuse`, `variable misuse`, `function misuse`])379- `failure_symptoms`: the problem the bug causes (one of [`incorrect output`, `stackoverflow`, `infinite loop`])380- `entry_point`: the name of the function381- `import`: imports necessary for the solution (only present for Go)382- `test_setup`: imports necessary for the test execution (only present for Go)383- `test`: the unit tests for the problem384- `example_test`: additional unit tests different from `test` that could be e.g. provided to the model (these are not used in the paper)385- `signature`: the signature of the function386- `docstring`: the docstring describing the problem387- `instruction`: an instruction for HumanEvalSynthesize in the form `Write a {language_name} function {signature} to solve the following problem:\n{docstring}`388 389## Citation Information390 391```bibtex392@article{muennighoff2023octopack,393      title={OctoPack: Instruction Tuning Code Large Language Models}, 394      author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},395      journal={arXiv preprint arXiv:2308.07124},396      year={2023}397}398```