e-zorzi/Qwen2.5-VL-7B-Instruct-tuned-raw
011
<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/> <details><summary>See axolotl config</summary>
axolotl version: 0.12.2
base_model: Qwen/Qwen2.5-VL-7B-Instruct
processor_type: AutoProcessor
# these 3 lines are needed for now to handle vision chat templates w images
skip_prepare_dataset: true
remove_unused_columns: false
sample_packing: false
chat_template: qwen2_vl
datasets:
- path: e-zorzi/reasoning_distractors_choice_chat
type: chat_template
split: train
test_datasets:
- path: e-zorzi/reasoning_distractors_choice_chat
type: chat_template
split: val_seen[:20%]
- path: e-zorzi/reasoning_distractors_choice_chat
type: chat_template
split: val_unseen[:20%]
output_dir: ../ctex-persistent/outputs/qwen2_5_VL_7B_lora
load_in_8bit: True
adapter: lora
lora_model_dir:
sequence_len: 2048 #8192
pad_to_sequence_len: false
lora_r: 128
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: 'model.language_model.layers.[\d]+.(mlp|cross_attn|self_attn).(up|down|gate|q|k|v|o)_proj'
wandb_project: axolotl_finetunes
wandb_entity: edo_vi
wandb_watch:
wandb_name: qwen_7B_2xH100
wandb_log_model:
gradient_accumulation_steps: 2
micro_batch_size: 32
num_epochs: 15
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.001
bf16: true
fp16:
tf32: true
gradient_checkpointing: true
logging_steps: 1
flash_attention: true
eager_attention:
warmup_steps: 60
evals_per_epoch: 2
saves_per_epoch: 1
save_strategy: epoch
weight_decay: 0.0
# save_first_step: true # uncomment this to validate checkpoint saving works with your config
</details><br>
ctex-persistent/outputs/qwen25VL7Blora
This model is a fine-tuned version of Qwen/Qwen2.5-VL-7B-Instruct on the e-zorzi/reasoningdistractorschoice_chat dataset. It achieves the following results on the evaluation set:
- Loss: 0.4372
- Memory/max Mem Active(gib): 74.46
- Memory/max Mem Allocated(gib): 74.46
- Memory/device Mem Reserved(gib): 77.0
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- trainbatchsize: 32
- evalbatchsize: 32
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradientaccumulationsteps: 2
- totaltrainbatch_size: 128
- totalevalbatch_size: 64
- optimizer: Use OptimizerNames.ADAMWBNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
- lrschedulertype: cosine
- lrschedulerwarmup_steps: 60
- training_steps: 1193
Training results
Framework versions
- PEFT 0.17.0
- Transformers 4.55.2
- Pytorch 2.6.0+cu126
- Datasets 4.0.0
- Tokenizers 0.21.4