Modelpublic

yalhessi/lemexp-task5-template_deriv_full-deepseek-coder-1.3b-base-4gpu-bs4-8lr-12epochs-normal-eos-8bit

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes6downloads

Model Card

lemexp-task5-templatederivfull-deepseek-coder-1.3b-base-4gpu-bs4-8lr-12epochs-normal-eos-8bit

This model is a fine-tuned version of deepseek-ai/deepseek-coder-1.3b-base on the None dataset. It achieves the following results on the evaluation set:

—Loss: 0.1664

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.0008
—trainbatchsize: 4
—evalbatchsize: 4
—seed: 42
—distributed_type: multi-GPU
—num_devices: 4
—totaltrainbatch_size: 16
—totalevalbatch_size: 16
—optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
—lrschedulertype: linear
—num_epochs: 12
—mixedprecisiontraining: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.5476	0.2001	720	0.3815
0.3642	0.4002	1440	0.3325
0.3018	0.6003	2160	0.3054
0.2847	0.8004	2880	0.2860
0.2684	1.0006	3600	0.2856
0.2595	1.2007	4320	0.2708
0.2471	1.4008	5040	0.2601
0.2465	1.6009	5760	0.2531
0.2403	1.8010	6480	0.2519
0.2324	2.0011	7200	0.2407
0.2247	2.2012	7920	0.2477
0.2178	2.4013	8640	0.2346
0.2171	2.6014	9360	0.2357
0.2173	2.8016	10080	0.2311
0.2153	3.0017	10800	0.2244
0.2006	3.2018	11520	0.2242
0.2002	3.4019	12240	0.2196
0.2012	3.6020	12960	0.2230
0.1987	3.8021	13680	0.2173
0.198	4.0022	14400	0.2114
0.1836	4.2023	15120	0.2114
0.1871	4.4024	15840	0.2061
0.1837	4.6026	16560	0.2070
0.1823	4.8027	17280	0.2032
0.1799	5.0028	18000	0.2007
0.167	5.2029	18720	0.2007
0.1716	5.4030	19440	0.1969
0.1682	5.6031	20160	0.1974
0.168	5.8032	20880	0.1974
0.1679	6.0033	21600	0.1864
0.1543	6.2034	22320	0.1903
0.156	6.4036	23040	0.1858
0.1569	6.6037	23760	0.1864
0.1538	6.8038	24480	0.1824
0.153	7.0039	25200	0.1845
0.1419	7.2040	25920	0.1857
0.1428	7.4041	26640	0.1766
0.1415	7.6042	27360	0.1749
0.1404	7.8043	28080	0.1750
0.1395	8.0044	28800	0.1734
0.1266	8.2046	29520	0.1728
0.1261	8.4047	30240	0.1724
0.1257	8.6048	30960	0.1695
0.1262	8.8049	31680	0.1710
0.1261	9.0050	32400	0.1683
0.112	9.2051	33120	0.1654
0.1107	9.4052	33840	0.1666
0.1111	9.6053	34560	0.1613
0.1104	9.8054	35280	0.1611
0.1106	10.0056	36000	0.1638
0.0952	10.2057	36720	0.1656
0.0981	10.4058	37440	0.1636
0.0978	10.6059	38160	0.1627
0.096	10.8060	38880	0.1627
0.0958	11.0061	39600	0.1639
0.0876	11.2062	40320	0.1684
0.0844	11.4063	41040	0.1679
0.0857	11.6064	41760	0.1669
0.0855	11.8066	42480	0.1664

Framework versions

—PEFT 0.14.0
—Transformers 4.47.0
—Pytorch 2.7.1+cu128
—Datasets 4.2.0
—Tokenizers 0.21.4