Salesforce/codet5-small

1---2license: apache-2.03tags:4- codet55datasets:6- code_search_net7inference: false8---9 10# CodeT5 (small-sized model) 11 12Pre-trained CodeT5 model. It was introduced in the paper [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models13for Code Understanding and Generation](https://arxiv.org/abs/2109.00859) by Yue Wang, Weishi Wang, Shafiq Joty, Steven C.H. Hoi and first released in [this repository](https://github.com/salesforce/CodeT5). 14 15Disclaimer: The team releasing CodeT5 did not write a model card for this model so this model card has been written by the Hugging Face team (more specifically, [nielsr](https://huggingface.co/nielsr)).16 17## Model description18 19From the abstract:20 21"We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code."22 23## Intended uses & limitations24 25This repository contains the pre-trained model only, so you can use this model for masked span prediction, as shown in the code example below. However, the main use of this model is to fine-tune it for a downstream task of interest, such as:26* code summarization27* code generation28* code translation29* code refinement30* code defect detection31* code clone detection. 32 33See the [model hub](https://huggingface.co/models?search=salesforce/codet) to look for fine-tuned versions on a task that interests you.34 35### How to use36 37Here is how to use this model:38 39```python40from transformers import RobertaTokenizer, T5ForConditionalGeneration41 42tokenizer = RobertaTokenizer.from_pretrained('Salesforce/codet5-small')43model = T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-small')44 45text = "def greet(user): print(f'hello <extra_id_0>!')"46input_ids = tokenizer(text, return_tensors="pt").input_ids47 48# simply generate a single sequence49generated_ids = model.generate(input_ids, max_length=10)50print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))51# this prints "user: {user.name}"52```53 54## Training data55 56The CodeT5 model was pretrained on CodeSearchNet [Husain et al., 2019](https://arxiv.org/abs/1909.09436). Additionally, the authors collected two datasets of C/CSharp from [BigQuery1](https://console.cloud.google.com/marketplace/details/github/github-repos) to ensure that all downstream tasks have overlapped programming languages with the pre-training data. In total, around 8.35 million instances are used for pretraining. 57 58## Training procedure59 60### Preprocessing61 62This model uses a code-specific BPE (Byte-Pair Encoding) tokenizer. One can prepare text (or code) for the model using RobertaTokenizer, with the files from this repository.63 64## Evaluation results65 66For evaluation results on several downstream benchmarks, we refer to the paper.67 68## Ethical Considerations69 70This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP. 71 72### BibTeX entry and citation info73 74```bibtex75@misc{wang2021codet5,76      title={CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation}, 77      author={Yue Wang and Weishi Wang and Shafiq Joty and Steven C. H. Hoi},78      year={2021},79      eprint={2109.00859},80      archivePrefix={arXiv},81      primaryClass={cs.CL}82}83```