CognitiveLab/Fireship-clone

1---2base_model: NousResearch/Llama-2-7b-hf3tags:4- generated_from_trainer5model-index:6- name: out7  results: []8---9 10```yaml11base_model: NousResearch/Llama-2-7b-hf12model_type: LlamaForCausalLM13tokenizer_type: LlamaTokenizer14is_llama_derived_model: true15 16load_in_8bit: false17load_in_4bit: false18strict: false19 20datasets:21  - path: CognitiveLab/FS_transcribe_summary_prompt22    type: completion23dataset_prepared_path: last_run_prepared24val_set_size: 0.0525output_dir: ./out26 27sequence_len: 409628sample_packing: true29pad_to_sequence_len: true30 31adapter:32lora_model_dir:33lora_r:34lora_alpha:35lora_dropout:36lora_target_linear:37lora_fan_in_fan_out:38 39wandb_project: fireship-fft40wandb_entity:41wandb_watch:42wandb_name:43wandb_log_model:44 45gradient_accumulation_steps: 446micro_batch_size: 447num_epochs: 148optimizer: adamw_bnb_8bit49lr_scheduler: cosine50learning_rate: 0.000251 52train_on_inputs: false53group_by_length: false54bf16: auto55fp16:56tf32: false57 58gradient_checkpointing: true59early_stopping_patience:60resume_from_checkpoint:61local_rank:62logging_steps: 163xformers_attention:64flash_attention: true65flash_attn_cross_entropy: false66flash_attn_rms_norm: true67flash_attn_fuse_qkv: false68flash_attn_fuse_mlp: true69 70warmup_steps: 10071evals_per_epoch: 472eval_table_size:73saves_per_epoch: 274debug:75deepspeed: #deepspeed_configs/zero2.json # multi-gpu only76weight_decay: 0.177fsdp:78fsdp_config:79special_tokens:80 81```82 83</details><br>84 85# out86 87This model is a fine-tuned version of [NousResearch/Llama-2-7b-hf](https://huggingface.co/NousResearch/Llama-2-7b-hf) on the None dataset.88It achieves the following results on the evaluation set:89- Loss: 1.844490 91## Model description92 93More information needed94 95## Intended uses & limitations96 97More information needed98 99## Training and evaluation data100 101More information needed102 103## Training procedure104 105### Training hyperparameters106 107The following hyperparameters were used during training:108- learning_rate: 0.0002109- train_batch_size: 4110- eval_batch_size: 4111- seed: 42112- gradient_accumulation_steps: 4113- total_train_batch_size: 16114- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08115- lr_scheduler_type: cosine116- lr_scheduler_warmup_steps: 100117- num_epochs: 1118 119### Training results120 121| Training Loss | Epoch | Step | Validation Loss |122|:-------------:|:-----:|:----:|:---------------:|123| 2.1256        | 0.06  | 1    | 2.1641          |124| 2.1049        | 0.25  | 4    | 2.1254          |125| 1.9826        | 0.49  | 8    | 1.9868          |126| 1.8545        | 0.74  | 12   | 1.8779          |127| 1.8597        | 0.98  | 16   | 1.8444          |128 129 130### Framework versions131 132- Transformers 4.36.2133- Pytorch 2.0.1+cu118134- Datasets 2.16.1135- Tokenizers 0.15.0136