Scale-or-Reason/gemma3-4B_0_split

1---2library_name: transformers3tags:4- generated_from_trainer5model-index:6- name: lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/07  results: []8---9 10<!-- This model card has been generated automatically according to the information the Trainer had access to. You11should probably proofread and complete it, then remove this comment. -->12 13[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)14<details><summary>See axolotl config</summary>15 16axolotl version: `0.12.2`17```yaml18base_model: /lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b19 20datasets:21- path: /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking22  ds_type: json23  type: chat_template24  field_messages: conversations25  data_files:26  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0007.jsonl27  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0009.jsonl28  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0005.jsonl29  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0006.jsonl30  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0014.jsonl31  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0010.jsonl32  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0012.jsonl33  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0008.jsonl34  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0001.jsonl35  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0002.jsonl36  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0013.jsonl37  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0015.jsonl38  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0004.jsonl39  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0011.jsonl40  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0000.jsonl41  - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0003.jsonl42 43dataset_prepared_path: /lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_044tokenizer_config: "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-27b"45chat_template: gemma346eot_tokens:47  - "<end_of_turn>"48 49shuffle_merged_datasets: true50output_dir: /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/051 52sequence_len: 1638453sample_packing: true54 55gradient_accumulation_steps: 156micro_batch_size: 157num_epochs: 0.658auto_resume_from_checkpoints: true59 60optimizer: adamw_torch_fused61lr_scheduler: warmup_stable_decay62learning_rate: 5e-663lr_scheduler_kwargs:64  num_decay_steps: 20065  min_lr_ratio: 0.166warmup_steps: 10067 68bf16: true69tf32: false70 71gradient_checkpointing: true72logging_steps: 1073flash_attention: true74 75evals_per_epoch: 076saves_per_epoch: 177save_total_limit: 2078save_only_model: true79 80use_tensorboard: true81deepspeed: /lustre/fswork/projects/rech/qwv/udv55np/axolotl/zero3.json82 83```84 85</details><br>86 87# lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/088 89This model was trained from scratch on the None dataset.90 91## Model description92 93More information needed94 95## Intended uses & limitations96 97More information needed98 99## Training and evaluation data100 101More information needed102 103## Training procedure104 105### Training hyperparameters106 107The following hyperparameters were used during training:108- learning_rate: 5e-06109- train_batch_size: 1110- eval_batch_size: 1111- seed: 42112- distributed_type: multi-GPU113- num_devices: 16114- total_train_batch_size: 16115- total_eval_batch_size: 16116- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments117- lr_scheduler_type: warmup_stable_decay118- lr_scheduler_warmup_steps: 100119- training_steps: 711120 121### Training results122 123 124 125### Framework versions126 127- Transformers 4.55.2128- Pytorch 2.6.0+cu124129- Datasets 4.0.0130- Tokenizers 0.21.1131