Modelpublic

qing-yao/babylm-opt-sva-decay-0.2-from8000-11473600

sourceHugging Faceupdated 16d agoView on Hugging Face

0likes830downloads

Model Card

babylm-opt-sva-decay-0.2-from8000-11473600

This model is a fine-tuned version of models/babylm-default_seed-42_1e-3 on the qing-yao/slightly-cleaner-babylm dataset. It achieves the following results on the evaluation set:

—Loss: 3.1216
—Num Input Tokens Seen: 1572314880

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.001
—trainbatchsize: 256
—evalbatchsize: 64
—seed: 42
—optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
—lrschedulertype: linear
—lrschedulerwarmup_steps: 32000
—num_epochs: 20.0
—mixedprecisiontraining: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
7.0783	0.1371	250	7.0315	16384000
5.3787	0.2741	500	5.3901	32768000
4.9499	0.4112	750	4.9824	49152000
4.8444	0.5482	1000	4.7563	65536000
4.6264	0.6853	1250	4.5572	81920000
4.4358	0.8224	1500	4.3957	98304000
4.3009	0.9594	1750	4.2668	114688000
4.1937	1.0965	2000	4.1604	131029760
4.0318	1.2336	2250	4.0687	147413760
4.0821	1.3706	2500	3.9881	163797760
4.0435	1.5077	2750	3.9142	180181760
3.8267	1.6447	3000	3.8492	196565760
3.8228	1.7818	3250	3.7909	212949760
3.8113	1.9189	3500	3.7387	229333760
3.6935	2.0559	3750	3.6923	245675520
3.7052	2.1930	4000	3.6491	262059520
3.6055	2.3300	4250	3.6096	278443520
3.616	2.4671	4500	3.5757	294827520
3.5368	2.6042	4750	3.5387	311211520
3.5042	2.7412	5000	3.5091	327595520
3.5091	2.8783	5250	3.4801	343979520
3.4449	3.0154	5500	3.4553	360321280
3.4085	3.1524	5750	3.4325	376705280
3.3963	3.2895	6000	3.4129	393089280
3.4197	3.4265	6250	3.3905	409473280
3.4021	3.5636	6500	3.3745	425857280
3.3401	3.7007	6750	3.3566	442241280
3.366	3.8377	7000	3.3367	458625280
3.3422	3.9748	7250	3.3252	475009280
3.2944	4.1118	7500	3.3128	491351040
3.4211	4.2489	7750	3.3001	507735040
3.2671	4.3860	8000	3.2890	524119040
3.3056	4.5230	8250	3.2779	540503040
3.2579	4.6601	8500	3.2685	556887040
3.1625	4.7971	8750	3.2596	573271040
3.2212	4.9342	9000	3.2495	589655040
3.1679	5.0713	9250	3.2421	605996800
3.2459	5.2083	9500	3.2378	622380800
3.1571	5.3454	9750	3.2312	638764800
3.1327	5.4825	10000	3.2236	655148800
3.1695	5.6195	10250	3.2176	671532800
3.2193	5.7566	10500	3.2104	687916800
3.179	5.8936	10750	3.2045	704300800
3.1305	6.0307	11000	3.2000	720642560
3.0805	6.1678	11250	3.1991	737026560
3.097	6.3048	11500	3.1949	753410560
3.1297	6.4419	11750	3.1897	769794560
3.0803	6.5789	12000	3.1859	786178560
3.1706	6.7160	12250	3.1771	802562560
3.1278	6.8531	12500	3.1724	818946560
3.113	6.9901	12750	3.1708	835330560
3.068	7.1272	13000	3.1712	851672320
3.0788	7.2643	13250	3.1707	868056320
3.0826	7.4013	13500	3.1642	884440320
3.0719	7.5384	13750	3.1616	900824320
3.1371	7.6754	14000	3.1566	917208320
3.0564	7.8125	14250	3.1553	933592320
3.0647	7.9496	14500	3.1513	949976320
3.0189	8.0866	14750	3.1553	966318080
3.0959	8.2237	15000	3.1542	982702080
3.0695	8.3607	15250	3.1500	999086080
3.1047	8.4978	15500	3.1461	1015470080
3.1008	8.6349	15750	3.1419	1031854080
3.0434	8.7719	16000	3.1376	1048238080
3.0703	8.9090	16250	3.1354	1064622080
3.0371	9.0461	16500	3.1421	1080963840
2.9787	9.1831	16750	3.1444	1097347840
2.9835	9.3202	17000	3.1398	1113731840
3.0261	9.4572	17250	3.1366	1130115840
3.0631	9.5943	17500	3.1328	1146499840
3.0508	9.7314	17750	3.1281	1162883840
3.0428	9.8684	18000	3.1277	1179267840
2.9921	10.0055	18250	3.1317	1195609600
2.9799	10.1425	18500	3.1360	1211993600
3.0509	10.2796	18750	3.1335	1228377600
2.9708	10.4167	19000	3.1295	1244761600
3.0124	10.5537	19250	3.1260	1261145600
3.0019	10.6908	19500	3.1220	1277529600
3.0757	10.8279	19750	3.1185	1293913600
3.0432	10.9649	20000	3.1145	1310297600
2.9169	11.1020	20250	3.1300	1326639360
2.9421	11.2390	20500	3.1277	1343023360
3.035	11.3761	20750	3.1237	1359407360
3.019	11.5132	21000	3.1205	1375791360
3.0319	11.6502	21250	3.1181	1392175360
3.0865	11.7873	21500	3.1175	1408559360
3.0199	11.9243	21750	3.1120	1424943360
2.9391	12.0614	22000	3.1250	1441285120
2.9815	12.1985	22250	3.1271	1457669120
3.0219	12.3355	22500	3.1219	1474053120
2.954	12.4726	22750	3.1186	1490437120
3.002	12.6096	23000	3.1155	1506821120
2.9974	12.7467	23250	3.1116	1523205120
3.0322	12.8838	23500	3.1082	1539589120
2.961	13.0208	23750	3.1172	1555930880
2.9561	13.1579	24000	3.1216	1572314880

Framework versions

—Transformers 4.49.0
—Pytorch 2.5.1+cu121
—Datasets 4.8.5
—Tokenizers 0.21.4