ibivibiv/llama3-8b-instruct-code

1---2license: llama33library_name: peft4tags:5- axolotl6- generated_from_trainer7base_model: meta-llama/Meta-Llama-3-8B-Instruct8model-index:9- name: llama3-8b-instruct-code10  results: []11language:12- en13---14 15<!-- This model card has been generated automatically according to the information the Trainer had access to. You16should probably proofread and complete it, then remove this comment. -->17 18[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)19<details><summary>See axolotl config</summary>20 21axolotl version: `0.4.0`22```yaml23adapter: qlora24base_model: meta-llama/Meta-Llama-3-8B-Instruct25base_model_config: meta-llama/Meta-Llama-3-8B-Instruct26datasets:27- path: candenizkocak/code-alpaca-297k28  type: alpaca29flash_attention: true30gradient_accumulation_steps: 431gradient_checkpointing: true32hf_use_auth_token: true33hub_model_id: ibivibiv/llama3-8b-instruct-code34learning_rate: 0.000235load_in_4bit: true36logging_steps: 137lora_alpha: 1638lora_dropout: 0.0539lora_r: 3240lora_target_linear: true41lr_scheduler: cosine42micro_batch_size: 243model_type: AutoModelForCausalLM44num_epochs: 345optimizer: paged_adamw_32bit46output_dir: /job/out47sample_packing: true48save_safetensors: true49sequence_len: 409650special_tokens:51  pad_token: <|end_of_text|>52tokenizer_type: AutoTokenizer53wandb_project: TuneStudio54wandb_run_id: codellama55wandb_watch: 'true'56warmup_steps: 1057 58```59 60</details><br>61 62# llama3-8b-instruct-code63 64This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on the None dataset.65 66## Model description67 68More information needed69 70## Intended uses & limitations71 72More information needed73 74## Training and evaluation data75 76More information needed77 78## Training procedure79 80### Training hyperparameters81 82The following hyperparameters were used during training:83- learning_rate: 0.000284- train_batch_size: 285- eval_batch_size: 286- seed: 4287- distributed_type: multi-GPU88- num_devices: 289- gradient_accumulation_steps: 490- total_train_batch_size: 1691- total_eval_batch_size: 492- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-0893- lr_scheduler_type: cosine94- lr_scheduler_warmup_steps: 1095- num_epochs: 396 97### Training results98 99 100 101### Framework versions102 103- PEFT 0.10.0104- Transformers 4.40.2105- Pytorch 2.1.2+cu118106- Datasets 2.19.1107- Tokenizers 0.19.1