SystemAdmin123/tiny-random-LlamaForCausalLM

1---2library_name: transformers3base_model: trl-internal-testing/tiny-random-LlamaForCausalLM4tags:5- axolotl6- generated_from_trainer7datasets:8- argilla/databricks-dolly-15k-curated-en9model-index:10- name: tiny-random-LlamaForCausalLM11  results: []12---13 14<!-- This model card has been generated automatically according to the information the Trainer had access to. You15should probably proofread and complete it, then remove this comment. -->16 17[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)18<details><summary>See axolotl config</summary>19 20axolotl version: `0.6.0`21```yaml22base_model: trl-internal-testing/tiny-random-LlamaForCausalLM23batch_size: 12824bf16: true25chat_template: tokenizer_default_fallback_alpaca26datasets:27- format: custom28  path: argilla/databricks-dolly-15k-curated-en29  type:30    field_input: original-instruction31    field_instruction: original-instruction32    field_output: original-response33    format: '{instruction} {input}'34    no_input_format: '{instruction}'35    system_format: '{system}'36    system_prompt: ''37device_map: auto38eval_sample_packing: false39eval_steps: 2040flash_attention: true41gradient_checkpointing: true42group_by_length: true43hub_model_id: SystemAdmin123/tiny-random-LlamaForCausalLM44hub_strategy: checkpoint45learning_rate: 0.000246logging_steps: 1047lr_scheduler: cosine48max_steps: 1000049micro_batch_size: 3250model_type: AutoModelForCausalLM51num_epochs: 10052optimizer: adamw_bnb_8bit53output_dir: /root/.sn56/axolotl/tmp/tiny-random-LlamaForCausalLM54pad_to_sequence_len: true55resize_token_embeddings_to_32x: false56sample_packing: true57save_steps: 2058save_total_limit: 159sequence_len: 204860tokenizer_type: LlamaTokenizerFast61torch_dtype: bf1662training_args_kwargs:63  hub_private_repo: true64trust_remote_code: true65val_set_size: 0.166wandb_entity: ''67wandb_mode: online68wandb_name: trl-internal-testing/tiny-random-LlamaForCausalLM-argilla/databricks-dolly-15k-curated-en69wandb_project: Gradients-On-Demand70wandb_run: your_name71wandb_runid: default72warmup_ratio: 0.0573 74```75 76</details><br>77 78# tiny-random-LlamaForCausalLM79 80This model is a fine-tuned version of [trl-internal-testing/tiny-random-LlamaForCausalLM](https://huggingface.co/trl-internal-testing/tiny-random-LlamaForCausalLM) on the argilla/databricks-dolly-15k-curated-en dataset.81It achieves the following results on the evaluation set:82- Loss: 10.181783 84## Model description85 86More information needed87 88## Intended uses & limitations89 90More information needed91 92## Training and evaluation data93 94More information needed95 96## Training procedure97 98### Training hyperparameters99 100The following hyperparameters were used during training:101- learning_rate: 0.0002102- train_batch_size: 32103- eval_batch_size: 32104- seed: 42105- distributed_type: multi-GPU106- num_devices: 4107- total_train_batch_size: 128108- total_eval_batch_size: 128109- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments110- lr_scheduler_type: cosine111- lr_scheduler_warmup_steps: 5112- training_steps: 100113 114### Training results115 116| Training Loss | Epoch   | Step | Validation Loss |117|:-------------:|:-------:|:----:|:---------------:|118| No log        | 0.1667  | 1    | 10.3764         |119| 10.3632       | 3.3333  | 20   | 10.3538         |120| 10.3073       | 6.6667  | 40   | 10.2840         |121| 10.2203       | 10.0    | 60   | 10.2082         |122| 10.1812       | 13.3333 | 80   | 10.1828         |123| 10.1767       | 16.6667 | 100  | 10.1817         |124 125 126### Framework versions127 128- Transformers 4.48.1129- Pytorch 2.5.1+cu124130- Datasets 3.2.0131- Tokenizers 0.21.0132