CoolFace
Modelpublic

allura-forge/qwen3.5-9b-seed2.0-claude-dist-v2-qlora-ckpts

sourceHugging Faceupdated 7mo agoView on Hugging Face
0likes35downloads
Model Card

output-fizz-v2

This model is a fine-tuned version of Qwen/Qwen3.5-9B.

W&B run: https://wandb.ai/cooawoo-personal/huggingface/runs/jelbo6gf

Training procedure

Hyperparameters

ParameterValue
Learning rate0.0002
LR schedulerSchedulerType.CONSTANT
Per-device batch size2
Gradient accumulation4
Effective batch size8
Epochs2
Max sequence length4096
OptimizerOptimizerNames.ADAMW_TORCH
Weight decay0.01
Warmup ratio0.05
Max gradient norm1.0
Precisionbf16
Loss typenll
Chunked cross-entropyyes

LoRA configuration

ParameterValue
Rank (r)64
Alpha512
Target modulesattn.proj, downproj, gateproj, inproja, inprojb, inprojqkv, inprojz, kproj, linearfc1, linearfc2, oproj, outproj, qproj, qkv, upproj, vproj
Quantization4-bit (nf4)

Dataset statistics

DatasetSamplesTotal tokensTrainable tokens
allura-forge/doubao-seed2.0-claude-distill-v1-qwen3.5-format3,6447,187,8566,625,087

<details> <summary>Training config</summary>

yaml
model_name_or_path: Qwen/Qwen3.5-9B
bf16: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
use_liger: true
use_cce: true
max_length: 4096
learning_rate: 0.0002
warmup_ratio: 0.05
weight_decay: 0.01
lr_scheduler_type: constant
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
optim: adamw_torch
max_grad_norm: 1.0
use_peft: true
load_in_4bit: true
lora_r: 64
lora_alpha: 512
lora_dropout: 0.0
use_rslora: false
logging_steps: 1
disable_tqdm: false
save_strategy: steps
save_steps: 500
save_total_limit: null
report_to: wandb
output_dir: output-fizz-v2
data_config: data.yaml
prepared_dataset: prepared
num_train_epochs: 2
saves_per_epoch: 1
run_name: qwen35-9b-qlora-fizz

</details>

<details> <summary>Data config</summary>

yaml
datasets:
- path: allura-forge/doubao-seed2.0-claude-distill-v1-qwen3.5-format
  type: conversational
  truncation_strategy: drop
shuffle_datasets: true
shuffle_combined: true
shuffle_seed: 42
eval_split: 0.0
split_seed: 42
assistant_only_loss: true

</details>

Framework versions

  • —PEFT 0.18.1
  • —Loft: 0.1.0
  • —Transformers: 5.2.0
  • —Pytorch: 2.6.0
  • —Datasets: 4.6.1
  • —Tokenizers: 0.22.2