web3se/SmartBERT-v3
5439
1---2license: mit3language:4- en5inference: true6base_model:7- microsoft/codebert-base-mlm8- web3se/SmartBERT-v29pipeline_tag: fill-mask10tags:11- fill-mask12- smart-contract13- web314- software-engineering15- embedding16- codebert17library_name: transformers18datasets:19- web3se/smart-contract-intent-vul-dataset20---21 22# SmartBERT V3 CodeBERT23 2425 26## Overview27 28**SmartBERT V3** is a domain-adapted pre-trained programming language model for **smart contract code understanding**, built upon **[CodeBERT-base-mlm](https://huggingface.co/microsoft/codebert-base-mlm)**.29 30The model is further trained on **[SmartBERT V2](https://huggingface.co/web3se/SmartBERT-v2)** with a substantially larger corpus of smart contracts, enabling improved robustness and richer semantic representations of **function-level smart contract code**.31 32SmartBERT V3 is particularly suitable for tasks such as:33 34- Smart contract intent detection35- Code representation learning36- Code similarity analysis37- Vulnerability detection38- Smart contract classification39 40Compared with **SmartBERT V2**, this version significantly expands the training corpus and improves the model’s ability to capture semantic patterns in smart contract functions.41 42---43 44## Training Data45 46SmartBERT V3 was trained on a total of **80,000 smart contracts**, including:47 48- **16,000 contracts** used in **[SmartBERT V2](https://huggingface.co/web3se/SmartBERT-v2)**49- **64,000 additional smart contracts** collected from public blockchain repositories50 51All contracts are primarily written in **Solidity** and processed at the **function level** to better capture fine-grained semantic structures of smart contract code.52 53---54 55## Training Objective56 57The model is trained using the **Masked Language Modeling (MLM)** objective, following the same training paradigm as **CodeBERT**.58 59During training:60 61- A subset of tokens in the input code is randomly masked62- The model learns to predict these masked tokens from surrounding context63 64This process enables the model to learn deeper **syntactic and semantic representations** of smart contract programs.65 66---67 68## Training Setup69 70Training was conducted using the **HuggingFace Transformers** framework.71 72- **Hardware:** 2 × Nvidia A100 (80GB)73- **Training Duration:** Over **30 hours**74- **Training Dataset:** 80,000 smart contracts75- **Evaluation Dataset:** 1,500 smart contracts76 77Example training configuration:78 79```python80training_args = TrainingArguments(81 output_dir=OUTPUT_DIR,82 overwrite_output_dir=True,83 num_train_epochs=20,84 per_device_train_batch_size=64,85 save_steps=10000,86 save_total_limit=2,87 evaluation_strategy="steps",88 eval_steps=10000,89 resume_from_checkpoint=checkpoint90)91````92 93---94 95## Preprocessing96 97During preprocessing, all newline (`\n`) and tab (`\t`) characters in the *function* code were replaced with a single space to ensure a consistent input format for tokenization.98 99---100 101## Base Model102 103SmartBERT V3 builds upon the following models:104 105* **Original Model**: [CodeBERT-base-mlm](https://huggingface.co/microsoft/codebert-base-mlm)106* **Intermediate Model**: [SmartBERT V2](https://huggingface.co/web3se/SmartBERT-v2)107 108---109 110## Usage111 112Example usage with HuggingFace Transformers:113 114```python115from transformers import RobertaTokenizer, RobertaForMaskedLM, pipeline116 117model = RobertaForMaskedLM.from_pretrained('web3se/SmartBERT-v3')118tokenizer = RobertaTokenizer.from_pretrained('web3se/SmartBERT-v3')119 120code_example = "function totalSupply() external view <mask> (uint256);"121fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)122 123outputs = fill_mask(code_example)124print(outputs)125```126 127---128 129## How to Use130 131To train and deploy the SmartBERT V3 model for Web API services, please refer to our GitHub repository: [web3se-lab/SmartBERT](https://github.com/web3se-lab/SmartBERT).132 133---134 135## Contributor136 137* [Youwei Huang](https://www.devil.ren)138* [Sen Fang](https://github.com/TomasAndersonFang)139 140---141 142## Citation143 144```tex145@article{huang2025smart,146 title={Smart Contract Intent Detection with Pre-trained Programming Language Model},147 author={Huang, Youwei and Li, Jianwen and Fang, Sen and Li, Yao and Yang, Peng and Hu, Bin},148 journal={arXiv preprint arXiv:2508.20086},149 year={2025}150}151```152 153---154 155## Acknowledgement156 157- [Institute of Intelligent Computing Technology, Suzhou, CAS](http://iict.ac.cn/)158- [Macau University of Science and Technology](http://www.must.edu.mo)159- CAS Mino (中科劢诺)