AlfredPros/CodeLlama-7b-Instruct-Solidity

1---2license: llama23datasets:4- AlfredPros/smart-contracts-instructions5language:6- en7tags:8- code9- blockchain10- solidity11- smart contract12---13# Code LLaMA 7B Instruct Solidity14 15A finetuned 7 billion parameters Code LLaMA - Instruct model to generate Solidity smart contract using 4-bit QLoRA finetuning provided by PEFT library.16 17# Training Dataset18 19Dataset used to finetune the model is AlfredPros' Smart Contracts Instructions (https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions). 20A dataset containing 6,003 GPT-generated human instruction and Solidity source code data pairs. This dataset has been processed for training LLMs.21 22# Training Parameters23 24## Bitsandbytes quantization configurations25- Load in 4-bit: true26- 4-bit quantization type: NF427- 4-bit compute dtype: float1628- 4-bit use double quantization: true29 30## Supervised finetuning trainer parameters31- Number of train epochs: 132- FP16: true33- FP16 option level: O134- BF16: false35- Per device train batch size: 136- Gradient accumulation steps: 137- Gradient checkpointing: true38- Max gradient normal: 0.339- Learning rate: 2e-440- Weight decay: 0.00141- Optimizer: paged AdamW 32-bit42- Learning rate scheduler type: cosine43- Warmup ratio: 0.0344 45# Training Details46- GPU used: 1x NVIDIA GeForce GTX 1080Ti47- Training time: 21 hours, 4 minutes, and 57 seconds48 49# Training Loss50```51Step	Training Loss52 100	0.33090053 200	0.29300054 300	0.27650055 400	0.29090056 500	0.30610057 600	0.30260058 700	0.33720059 800	0.29500060 900	0.297800611000	0.299500621100	0.268900631200	0.257800641300	0.264100651400	0.294400661500	0.293900671600	0.287600681700	0.281200691800	0.273400701900	0.266600712000	0.227500722100	0.261600732200	0.275700742300	0.290100752400	0.290900762500	0.316200772600	0.296500782700	0.291400792800	0.253300802900	0.321500813000	0.269500823100	0.295600833200	0.265800843300	0.262800853400	0.274900863500	0.259800873600	0.226300883700	0.325700893800	0.249000903900	0.237200914000	0.251400924100	0.247000934200	0.278700944300	0.264000954400	0.245000964500	0.235900974600	0.240400984700	0.235200994800	0.2203001004900	0.2027001015000	0.2405001025100	0.2585001035200	0.2363001045300	0.2675001055400	0.2367001065500	0.2659001075600	0.2449001085700	0.2979001095800	0.2812001105900	0.3138001116000	0.2498001126003	0.271939113```114 115# Example Usage116```py117from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM118import torch119import accelerate120 121use_4bit = True122bnb_4bit_compute_dtype = "float16"123bnb_4bit_quant_type = "nf4"124use_double_nested_quant = True125compute_dtype = getattr(torch, bnb_4bit_compute_dtype)126 127# BitsAndBytesConfig 4-bit config128bnb_config = BitsAndBytesConfig(129    load_in_4bit=use_4bit,130    bnb_4bit_use_double_quant=use_double_nested_quant,131    bnb_4bit_quant_type=bnb_4bit_quant_type,132    bnb_4bit_compute_dtype=compute_dtype,133    load_in_8bit_fp32_cpu_offload=True134)135 136# Load model in 4-bit137tokenizer = AutoTokenizer.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity")138model = AutoModelForCausalLM.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity", quantization_config=bnb_config, device_map="balanced_low_0")139 140# Make input141input='Make a smart contract to create a whitelist of approved wallets. The purpose of this contract is to allow the DAO (Decentralized Autonomous Organization) to approve or revoke certain wallets, and also set a checker address for additional validation if needed. The current owner address can be changed by the current owner.'142 143# Make prompt template144prompt = f"""### Instruction:145Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:146 147### Task:148{input}149 150### Solution:151"""152 153# Tokenize the input154input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()155# Run the model to infere an output156outputs = model.generate(input_ids=input_ids, max_new_tokens=1024, do_sample=True, top_p=0.9, temperature=0.001, pad_token_id=1)157 158# Detokenize and display the generated output159print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):])160```