Modelpublic

philipperen55/Qwen2.5-14B-datasetSFT73_lora3

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes5downloads

Model Card

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/> <details><summary>See axolotl config</summary>

axolotl version: 0.17.0.dev0

yaml

base_model: Qwen/Qwen2.5-14B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

load_in_8bit: false
load_in_4bit: false


datasets:
  - path: philipperen55/datasetSFT73 
    ds_type: json
    data_files: datasetSFT73.jsonl
    type: input_output
  

train_on_inputs: false
add_eos_token: false

  
dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.05
output_dir: /workspace/output


sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
group_by_length: true



special_tokens:
  eos_token: "<|endoftext|>"
  pad_token: "<|endoftext|>"


adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.1
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj


gradient_accumulation_steps: 4
micro_batch_size: 4 
num_epochs: 3
learning_rate: 5e-5
lr_scheduler: cosine
warmup_ratio: 0.05
optimizer: adamw_torch_fused
weight_decay: 0.01
max_grad_norm: 1.0


bf16: true
fp16: false
tf32: true


attn_implementation: flash_attention_2


overrides_of_model_config:
  use_cache: false
  

gradient_checkpointing: false


seed: 42
#mettre 24 si ya plus de 24 vspu, sinon mettre 16 si ya 24vcpu
dataset_num_proc: 16


logging_steps: 5
save_steps: 50
eval_strategy: steps 
eval_steps: 50
save_total_limit: 1


wandb_project: datasetSFT73


hub_model_id: philipperen55/Qwen2.5-14B-datasetSFT73_lora3
push_to_hub: true
hub_strategy: every_save

</details><br>

Qwen2.5-14B-datasetSFT73_lora3

This model is a fine-tuned version of Qwen/Qwen2.5-14B on the philipperen55/datasetSFT73 dataset. It achieves the following results on the evaluation set:

—Loss: 1.5650
—Ppl: 4.7826
—Memory/max Active (gib): 122.64
—Memory/max Allocated (gib): 122.64
—Memory/device Reserved (gib): 155.38

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 5e-05
—trainbatchsize: 4
—evalbatchsize: 4
—seed: 42
—gradientaccumulationsteps: 4
—totaltrainbatch_size: 16
—optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
—lrschedulertype: cosine
—lrschedulerwarmup_steps: 14
—training_steps: 288

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	1.7502	5.7559	38.17	38.17	45.57
1.5393	0.5208	50	1.6318	5.1129	39.21	39.21	146.52
1.5068	1.0417	100	1.5909	4.9080	39.21	39.21	173.35
1.5239	1.5625	150	1.5747	4.8291	39.21	39.21	148.14
1.5132	2.0833	200	1.5735	4.8236	39.21	39.21	157.84
1.5752	2.6042	250	1.5650	4.7826	39.21	39.21	160.44
1.2400	3.0	288	1.5650	4.7826	122.64	122.64	155.38

Framework versions

—PEFT 0.19.1
—Transformers 5.9.0
—Pytorch 2.10.0+cu128
—Datasets 4.8.5
—Tokenizers 0.22.2