felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-0p5-token-diff-ctx5-v2
018
<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/> <details><summary>See axolotl config</summary>
axolotl version: 0.13.2
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
model_type: Qwen2ForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
# Pre-tokenized datasets produced by scripts/preprocess_diff_mask_chat.py
# (from felixwangg/prime_vul_plus_splitted which has a fixed validation split).
# Columns: input_ids, attention_mask, labels, diff_mask.
# Labels are already -100 for non-assistant tokens; axolotl keeps them as-is.
datasets:
- path: felixwangg/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx5_chat_v2
type: pretokenized
split: train
test_datasets:
- path: felixwangg/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx5_chat_v2
type: pretokenized
split: validation
dataset_prepared_path: /scratch/tkwang/SecSteer/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx5_chat_v2_alpha_0p5
val_set_size: 0
output_dir: /scratch/tkwang/SecSteer/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-0p5-token-diff-ctx5-v2
sequence_len: 4096
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: true
adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
merge_lora: true
wandb_project: diff-mask-sft-primevul-ctx-5
wandb_entity: wtkuan
wandb_watch: "false"
wandb_name: Qwen2.5-Coder-7B-sft-plus-alpha-0p5-token-diff-ctx5-v2
wandb_log_model: "false"
gradient_accumulation_steps: 8
micro_batch_size: 4
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 4e-05
bf16: true
tf32: false
gradient_checkpointing: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: true
num_epochs: 1
warmup_ratio: 0.1
early_stopping_patience: 1000
eval_steps: 15
save_steps: 15
save_total_limit: 1000
load_best_model_at_end: true
weight_decay: 0.02
special_tokens:
# Diff-mask weighted loss: CE(logit_t, label_t) * (1 + alpha * diff_mask_{t+1})
# Security-sensitive tokens (diff_mask=1) get weight (1 + diff_mask_alpha).
# Requires PYTHONPATH to include the repo root so diff_mask_trainer is importable.
diff_mask_alpha: 0.5
plugins:
- diff_mask_trainer.plugin.DiffMaskPlugin
# - sec_bench_callback.SecBenchPlugin
</details><br>
scratch/tkwang/SecSteer/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-0p5-token-diff-ctx5-v2
This model is a fine-tuned version of Qwen/Qwen2.5-Coder-7B-Instruct on the felixwangg/primevulplussplittedtokendiffmaskskipindentctx5chat_v2 dataset. It achieves the following results on the evaluation set:
- Loss: 0.7682
- Ppl: 2.1559
- Memory/max Active (gib): 42.7
- Memory/max Allocated (gib): 42.7
- Memory/device Reserved (gib): 61.76
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 4e-05
- trainbatchsize: 4
- evalbatchsize: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradientaccumulationsteps: 8
- totaltrainbatch_size: 64
- totalevalbatch_size: 8
- optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
- lrschedulertype: cosine
- lrschedulerwarmup_steps: 5
- training_steps: 57
Training results
Framework versions
- PEFT 0.18.1
- Transformers 4.57.6
- Pytorch 2.10.0+cu128
- Datasets 4.5.0
- Tokenizers 0.22.2