Modelpublic

nbeerbower/Hemlock-Qwen2.5-Coder-14B

sourceHugging Faceupdated 9mo agoView on Hugging Face

1likes20downloads

Model Card

image/png

Hemlock-Qwen2.5-Coder-14B

An attempt to train Qwen2.5-Coder on The Hemlock Programming Language using SFT.

Parameter	Value
Training Mode	SFT
Base Model	`Qwen/Qwen2.5-Coder-14B-Instruct`
Learning Rate	9e-05
Epochs	2
Batch Size	1
Gradient Accumulation	16
Effective Batch Size	16
Max Sequence Length	2048
Optimizer	pagedadamw8bit
LR Scheduler	cosine
Warmup Ratio	0.05
Weight Decay	0.01
Max Grad Norm	0.25
Seed	42
LoRA Rank (r)	128
LoRA Alpha	64
LoRA Dropout	0.05
Target Modules	upproj, downproj, gateproj, kproj, qproj, vproj, o_proj
Quantization	4-bit (NF4)
GPU	NVIDIA RTX A6000

Trained with Merlina