Modelpublic

flammenai/Phoenix-Qwen2.5-3B-v1

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes13downloads

Model Card

Phoenix-Qwen2.5-3B-v1

Parameter	Value
Training Mode	SFT
Base Model	`Qwen/Qwen2.5-3B-Instruct`
Learning Rate	0.0001
Epochs	2
Batch Size	1
Gradient Accumulation	8
Effective Batch Size	8
Max Sequence Length	2048
Optimizer	pagedadamw8bit
LR Scheduler	cosine
Warmup Ratio	0.05
Weight Decay	0.01
Max Grad Norm	0.5
Seed	42
LoRA Rank (r)	128
LoRA Alpha	128
LoRA Dropout	0.05
Target Modules	upproj, downproj, gateproj, kproj, qproj, vproj, o_proj
Quantization	4-bit (NF4)
GPU	NVIDIA RTX A6000

Trained with Merlina