peterhung/EvolCodeLlama-7b

1---2license: llama23library_name: peft4tags:5- axolotl6- generated_from_trainer7base_model: codellama/CodeLlama-7b-hf8model-index:9- name: EvolCodeLlama-7b10  results: []11---12 13<!-- This model card has been generated automatically according to the information the Trainer had access to. You14should probably proofread and complete it, then remove this comment. -->15 16[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)17<details><summary>See axolotl config</summary>18 19axolotl version: `0.4.0`20```yaml21base_model: codellama/CodeLlama-7b-hf22base_model_config: codellama/CodeLlama-7b-hf23model_type: LlamaForCausalLM24tokenizer_type: LlamaTokenizer25is_llama_derived_model: true26hub_model_id: EvolCodeLlama-7b27 28load_in_8bit: false29load_in_4bit: true30strict: false31 32datasets:33  - path: mlabonne/Evol-Instruct-Python-1k34    type: alpaca35dataset_prepared_path: last_run_prepared36val_set_size: 0.0237output_dir: ./qlora-out38 39adapter: qlora40lora_model_dir:41 42sequence_len: 204843sample_packing: true44 45lora_r: 3246lora_alpha: 1647lora_dropout: 0.0548lora_target_modules:49lora_target_linear: true50lora_fan_in_fan_out:51 52wandb_project: FTCodeLlama-253wandb_entity:54wandb_watch:55wandb_run_id:56wandb_log_model:57 58gradient_accumulation_steps: 259micro_batch_size: 460num_epochs: 361optimizer: paged_adamw_32bit62lr_scheduler: cosine63learning_rate: 0.000264 65train_on_inputs: false66group_by_length: false67bf16: true68fp16: false69tf32: false70 71gradient_checkpointing: true72early_stopping_patience:73resume_from_checkpoint:74local_rank:75logging_steps: 176xformers_attention:77flash_attention: true78 79warmup_steps: 10080eval_steps: 0.0181save_strategy: epoch82save_steps:83debug:84deepspeed:85weight_decay: 0.086fsdp:87fsdp_config:88special_tokens:89  bos_token: "<s>"90  eos_token: "</s>"91  unk_token: "<unk>"92 93```94 95</details><br>96 97# EvolCodeLlama-7b98 99This model is a fine-tuned version of [codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf) on the None dataset.100It achieves the following results on the evaluation set:101- Loss: 0.3754102 103## Model description104 105More information needed106 107## Intended uses & limitations108 109More information needed110 111## Training and evaluation data112 113More information needed114 115## Training procedure116 117### Training hyperparameters118 119The following hyperparameters were used during training:120- learning_rate: 0.0002121- train_batch_size: 4122- eval_batch_size: 4123- seed: 42124- gradient_accumulation_steps: 2125- total_train_batch_size: 8126- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08127- lr_scheduler_type: cosine128- lr_scheduler_warmup_steps: 100129- num_epochs: 3130 131### Training results132 133| Training Loss | Epoch | Step | Validation Loss |134|:-------------:|:-----:|:----:|:---------------:|135| 0.3686        | 0.01  | 1    | 0.5015          |136| 0.4397        | 0.03  | 3    | 0.5013          |137| 0.4919        | 0.06  | 6    | 0.5013          |138| 0.3191        | 0.09  | 9    | 0.5011          |139| 0.2514        | 0.12  | 12   | 0.5003          |140| 0.3379        | 0.15  | 15   | 0.4992          |141| 0.4712        | 0.19  | 18   | 0.4969          |142| 0.3801        | 0.22  | 21   | 0.4922          |143| 0.3482        | 0.25  | 24   | 0.4833          |144| 0.4113        | 0.28  | 27   | 0.4702          |145| 0.2524        | 0.31  | 30   | 0.4552          |146| 0.2641        | 0.34  | 33   | 0.4415          |147| 0.3554        | 0.37  | 36   | 0.4302          |148| 0.2384        | 0.4   | 39   | 0.4213          |149| 0.2131        | 0.43  | 42   | 0.4153          |150| 0.2308        | 0.46  | 45   | 0.4105          |151| 0.3478        | 0.49  | 48   | 0.4053          |152| 0.2913        | 0.53  | 51   | 0.4003          |153| 0.2909        | 0.56  | 54   | 0.3956          |154| 0.2032        | 0.59  | 57   | 0.3928          |155| 0.2479        | 0.62  | 60   | 0.3906          |156| 0.2145        | 0.65  | 63   | 0.3890          |157| 0.2447        | 0.68  | 66   | 0.3882          |158| 0.2928        | 0.71  | 69   | 0.3876          |159| 0.384         | 0.74  | 72   | 0.3854          |160| 0.1751        | 0.77  | 75   | 0.3835          |161| 0.352         | 0.8   | 78   | 0.3818          |162| 0.2443        | 0.84  | 81   | 0.3808          |163| 0.3211        | 0.87  | 84   | 0.3798          |164| 0.3026        | 0.9   | 87   | 0.3788          |165| 0.2357        | 0.93  | 90   | 0.3776          |166| 0.2661        | 0.96  | 93   | 0.3755          |167| 0.3314        | 0.99  | 96   | 0.3751          |168| 0.2789        | 1.02  | 99   | 0.3742          |169| 0.1734        | 1.03  | 102  | 0.3744          |170| 0.1928        | 1.06  | 105  | 0.3761          |171| 0.2681        | 1.09  | 108  | 0.3753          |172| 0.4148        | 1.12  | 111  | 0.3750          |173| 0.1977        | 1.15  | 114  | 0.3744          |174| 0.1977        | 1.19  | 117  | 0.3740          |175| 0.2499        | 1.22  | 120  | 0.3742          |176| 0.2192        | 1.25  | 123  | 0.3730          |177| 0.2207        | 1.28  | 126  | 0.3723          |178| 0.2179        | 1.31  | 129  | 0.3718          |179| 0.2843        | 1.34  | 132  | 0.3734          |180| 0.2614        | 1.37  | 135  | 0.3721          |181| 0.2033        | 1.4   | 138  | 0.3705          |182| 0.212         | 1.43  | 141  | 0.3705          |183| 0.2307        | 1.46  | 144  | 0.3712          |184| 0.3182        | 1.49  | 147  | 0.3698          |185| 0.2467        | 1.53  | 150  | 0.3664          |186| 0.1909        | 1.56  | 153  | 0.3665          |187| 0.3286        | 1.59  | 156  | 0.3655          |188| 0.2195        | 1.62  | 159  | 0.3648          |189| 0.3231        | 1.65  | 162  | 0.3650          |190| 0.2922        | 1.68  | 165  | 0.3631          |191| 0.1956        | 1.71  | 168  | 0.3627          |192| 0.2299        | 1.74  | 171  | 0.3639          |193| 0.1585        | 1.77  | 174  | 0.3649          |194| 0.2289        | 1.8   | 177  | 0.3650          |195| 0.189         | 1.84  | 180  | 0.3643          |196| 0.2736        | 1.87  | 183  | 0.3628          |197| 0.3591        | 1.9   | 186  | 0.3614          |198| 0.3181        | 1.93  | 189  | 0.3612          |199| 0.1994        | 1.96  | 192  | 0.3612          |200| 0.2499        | 1.99  | 195  | 0.3618          |201| 0.1659        | 2.01  | 198  | 0.3627          |202| 0.231         | 2.04  | 201  | 0.3665          |203| 0.169         | 2.07  | 204  | 0.3744          |204| 0.2082        | 2.1   | 207  | 0.3800          |205| 0.1755        | 2.13  | 210  | 0.3770          |206| 0.1959        | 2.16  | 213  | 0.3721          |207| 0.1933        | 2.19  | 216  | 0.3705          |208| 0.1213        | 2.22  | 219  | 0.3712          |209| 0.237         | 2.25  | 222  | 0.3738          |210| 0.2277        | 2.28  | 225  | 0.3771          |211| 0.2832        | 2.31  | 228  | 0.3789          |212| 0.2039        | 2.35  | 231  | 0.3783          |213| 0.2302        | 2.38  | 234  | 0.3764          |214| 0.1562        | 2.41  | 237  | 0.3750          |215| 0.1688        | 2.44  | 240  | 0.3742          |216| 0.126         | 2.47  | 243  | 0.3741          |217| 0.1846        | 2.5   | 246  | 0.3746          |218| 0.2195        | 2.53  | 249  | 0.3745          |219| 0.2335        | 2.56  | 252  | 0.3749          |220| 0.1542        | 2.59  | 255  | 0.3750          |221| 0.1783        | 2.62  | 258  | 0.3755          |222| 0.2409        | 2.65  | 261  | 0.3762          |223| 0.1777        | 2.69  | 264  | 0.3762          |224| 0.2591        | 2.72  | 267  | 0.3761          |225| 0.2092        | 2.75  | 270  | 0.3757          |226| 0.2256        | 2.78  | 273  | 0.3757          |227| 0.1923        | 2.81  | 276  | 0.3756          |228| 0.156         | 2.84  | 279  | 0.3755          |229| 0.2036        | 2.87  | 282  | 0.3754          |230| 0.2254        | 2.9   | 285  | 0.3753          |231| 0.1683        | 2.93  | 288  | 0.3753          |232| 0.1528        | 2.96  | 291  | 0.3754          |233 234 235### Framework versions236 237- PEFT 0.8.2238- Transformers 4.38.0.dev0239- Pytorch 2.1.2+cu118240- Datasets 2.17.0241- Tokenizers 0.15.0