RLAIF/mbpp

1---2annotations_creators:3- crowdsourced4- expert-generated5language_creators:6- crowdsourced7- expert-generated8language:9- en10license:11- cc-by-4.012multilinguality:13- monolingual14size_categories:15- n<1K16source_datasets:17- original18task_categories:19- text2text-generation20task_ids: []21pretty_name: Mostly Basic Python Problems22tags:23- code-generation24dataset_info:25- config_name: full26  features:27  - name: task_id28    dtype: int3229  - name: text30    dtype: string31  - name: code32    dtype: string33  - name: test_list34    sequence: string35  - name: test_setup_code36    dtype: string37  - name: challenge_test_list38    sequence: string39  splits:40  - name: train41    num_bytes: 17687942    num_examples: 37443  - name: test44    num_bytes: 24410445    num_examples: 50046  - name: validation47    num_bytes: 4240548    num_examples: 9049  - name: prompt50    num_bytes: 455051    num_examples: 1052  download_size: 23606953  dataset_size: 46793854- config_name: sanitized55  features:56  - name: source_file57    dtype: string58  - name: task_id59    dtype: int3260  - name: prompt61    dtype: string62  - name: code63    dtype: string64  - name: test_imports65    sequence: string66  - name: test_list67    sequence: string68  splits:69  - name: train70    num_bytes: 6345371    num_examples: 12072  - name: test73    num_bytes: 13272074    num_examples: 25775  - name: validation76    num_bytes: 2005077    num_examples: 4378  - name: prompt79    num_bytes: 340780    num_examples: 781  download_size: 11542282  dataset_size: 21963083configs:84- config_name: full85  data_files:86  - split: train87    path: full/train-*88  - split: test89    path: full/test-*90  - split: validation91    path: full/validation-*92  - split: prompt93    path: full/prompt-*94  default: true95- config_name: sanitized96  data_files:97  - split: train98    path: sanitized/train-*99  - split: test100    path: sanitized/test-*101  - split: validation102    path: sanitized/validation-*103  - split: prompt104    path: sanitized/prompt-*105---106 107# Dataset Card for Mostly Basic Python Problems (mbpp)108 109## Table of Contents110- [Dataset Card for Mostly Basic Python Problems (mbpp)](#dataset-card-for-mostly-basic-python-problems-(mbpp))111  - [Table of Contents](#table-of-contents)112  - [Dataset Description](#dataset-description)113    - [Dataset Summary](#dataset-summary)114    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)115    - [Languages](#languages)116  - [Dataset Structure](#dataset-structure)117    - [Data Instances](#data-instances)118    - [Data Fields](#data-fields)119    - [Data Splits](#data-splits)120  - [Dataset Creation](#dataset-creation)121    - [Curation Rationale](#curation-rationale)122    - [Source Data](#source-data)123      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)124      - [Who are the source language producers?](#who-are-the-source-language-producers)125    - [Annotations](#annotations)126      - [Annotation process](#annotation-process)127      - [Who are the annotators?](#who-are-the-annotators)128    - [Personal and Sensitive Information](#personal-and-sensitive-information)129  - [Considerations for Using the Data](#considerations-for-using-the-data)130    - [Social Impact of Dataset](#social-impact-of-dataset)131    - [Discussion of Biases](#discussion-of-biases)132    - [Other Known Limitations](#other-known-limitations)133  - [Additional Information](#additional-information)134    - [Dataset Curators](#dataset-curators)135    - [Licensing Information](#licensing-information)136    - [Citation Information](#citation-information)137    - [Contributions](#contributions)138 139## Dataset Description140- **Repository:** https://github.com/google-research/google-research/tree/master/mbpp141- **Paper:** [Program Synthesis with Large Language Models](https://arxiv.org/abs/2108.07732)142 143### Dataset Summary144The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by us. 145 146Released [here](https://github.com/google-research/google-research/tree/master/mbpp) as part of [Program Synthesis with Large Language Models, Austin et. al., 2021](https://arxiv.org/abs/2108.07732).147 148### Supported Tasks and Leaderboards149This dataset is used to evaluate code generations.150 151### Languages152English - Python code153 154## Dataset Structure155 156```python157dataset_full = load_dataset("mbpp")158DatasetDict({159    test: Dataset({160        features: ['task_id', 'text', 'code', 'test_list', 'test_setup_code', 'challenge_test_list'],161        num_rows: 974162    })163})164 165dataset_sanitized = load_dataset("mbpp", "sanitized")166DatasetDict({167    test: Dataset({168        features: ['source_file', 'task_id', 'prompt', 'code', 'test_imports', 'test_list'],169        num_rows: 427170    })171})172```173 174### Data Instances175 176#### mbpp - full177```178{179    'task_id': 1,180    'text': 'Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix cost[][] and a position (m, n) in cost[][].',181    'code': 'R = 3\r\nC = 3\r\ndef min_cost(cost, m, n): \r\n\ttc = [[0 for x in range(C)] for x in range(R)] \r\n\ttc[0][0] = cost[0][0] \r\n\tfor i in range(1, m+1): \r\n\t\ttc[i][0] = tc[i-1][0] + cost[i][0] \r\n\tfor j in range(1, n+1): \r\n\t\ttc[0][j] = tc[0][j-1] + cost[0][j] \r\n\tfor i in range(1, m+1): \r\n\t\tfor j in range(1, n+1): \r\n\t\t\ttc[i][j] = min(tc[i-1][j-1], tc[i-1][j], tc[i][j-1]) + cost[i][j] \r\n\treturn tc[m][n]',182    'test_list': [183        'assert min_cost([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2) == 8',184        'assert min_cost([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2) == 12',185        'assert min_cost([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2) == 16'],186    'test_setup_code': '',187    'challenge_test_list': []188}189```190#### mbpp - sanitized191```192{193    'source_file': 'Benchmark Questions Verification V2.ipynb',194    'task_id': 2,195    'prompt': 'Write a function to find the shared elements from the given two lists.',196    'code': 'def similar_elements(test_tup1, test_tup2):\n  res = tuple(set(test_tup1) & set(test_tup2))\n  return (res) ',197    'test_imports': [],198    'test_list': [199        'assert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))',200        'assert set(similar_elements((1, 2, 3, 4),(5, 4, 3, 7))) == set((3, 4))',201        'assert set(similar_elements((11, 12, 14, 13),(17, 15, 14, 13))) == set((13, 14))'202        ]203}204```205### Data Fields206 207- `source_file`: unknown208- `text`/`prompt`: description of programming task209- `code`: solution for programming task210- `test_setup_code`/`test_imports`: necessary code imports to execute tests211- `test_list`: list of tests to verify solution212- `challenge_test_list`: list of more challenging test to further probe solution213 214### Data Splits215There are two version of the dataset (full and sanitized), each with four splits:216- train217- evaluation218- test219- prompt220 221The `prompt` split corresponds to samples used for few-shot prompting and not for training.222 223## Dataset Creation224See section 2.1 of original [paper](https://arxiv.org/abs/2108.07732).225 226### Curation Rationale227In order to evaluate code generation functions a set of simple programming tasks as well as solutions is necessary which this dataset provides.228 229### Source Data230 231#### Initial Data Collection and Normalization232The dataset was manually created from scratch.233 234#### Who are the source language producers?235The dataset was created with an internal crowdsourcing effort at Google.236 237### Annotations238 239#### Annotation process240The full dataset was created first and a subset then underwent a second round to improve the task descriptions.241 242#### Who are the annotators?243The dataset was created with an internal crowdsourcing effort at Google.244 245### Personal and Sensitive Information246None.247 248## Considerations for Using the Data249Make sure you execute generated Python code in a safe environment when evauating against this dataset as generated code could be harmful.250 251### Social Impact of Dataset252With this dataset code generating models can be better evaluated which leads to fewer issues introduced when using such models.253 254### Discussion of Biases255 256### Other Known Limitations257Since the task descriptions might not be expressive enough to solve the task. The `sanitized` split aims at addressing this issue by having a second round of annotators improve the dataset.258 259## Additional Information260 261### Dataset Curators262Google Research263 264### Licensing Information265CC-BY-4.0266 267### Citation Information268```269@article{austin2021program,270  title={Program Synthesis with Large Language Models},271  author={Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others},272  journal={arXiv preprint arXiv:2108.07732},273  year={2021}274```275### Contributions276Thanks to [@lvwerra](https://github.com/lvwerra) for adding this dataset.