Modelpublic

qing-yao/babylm-opt-sva-seed42-11451088

sourceHugging Faceupdated 16d agoView on Hugging Face

0likes1.1kdownloads

Model Card

babylm-opt-sva-seed42-11451088

This model is a fine-tuned version of models/babylm-default_seed-42_1e-3 on the qing-yao/slightly-cleaner-babylm dataset. It achieves the following results on the evaluation set:

—Loss: 2.9999
—Num Input Tokens Seen: 2389908480

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.001
—trainbatchsize: 256
—evalbatchsize: 64
—seed: 42
—optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
—lrschedulertype: linear
—lrschedulerwarmup_steps: 32000
—num_epochs: 20.0
—mixedprecisiontraining: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
7.0783	0.1371	250	7.0315	16384000
5.3787	0.2741	500	5.3901	32768000
4.9499	0.4112	750	4.9824	49152000
4.8444	0.5482	1000	4.7563	65536000
4.6264	0.6853	1250	4.5572	81920000
4.4358	0.8224	1500	4.3957	98304000
4.3009	0.9594	1750	4.2668	114688000
4.1937	1.0965	2000	4.1604	131029760
4.0318	1.2336	2250	4.0687	147413760
4.0821	1.3706	2500	3.9881	163797760
4.0435	1.5077	2750	3.9142	180181760
3.8267	1.6447	3000	3.8492	196565760
3.8228	1.7818	3250	3.7909	212949760
3.8113	1.9189	3500	3.7387	229333760
3.6935	2.0559	3750	3.6923	245675520
3.7052	2.1930	4000	3.6491	262059520
3.6055	2.3300	4250	3.6096	278443520
3.616	2.4671	4500	3.5757	294827520
3.5368	2.6042	4750	3.5387	311211520
3.5042	2.7412	5000	3.5091	327595520
3.5091	2.8783	5250	3.4801	343979520
3.4449	3.0154	5500	3.4553	360321280
3.4085	3.1524	5750	3.4325	376705280
3.3963	3.2895	6000	3.4129	393089280
3.4197	3.4265	6250	3.3905	409473280
3.4021	3.5636	6500	3.3745	425857280
3.3401	3.7007	6750	3.3566	442241280
3.366	3.8377	7000	3.3367	458625280
3.3422	3.9748	7250	3.3252	475009280
3.2944	4.1118	7500	3.3128	491351040
3.4211	4.2489	7750	3.3001	507735040
3.2671	4.3860	8000	3.2890	524119040
3.3047	4.5230	8250	3.2775	540503040
3.2559	4.6601	8500	3.2669	556887040
3.1606	4.7971	8750	3.2583	573271040
3.2175	4.9342	9000	3.2468	589655040
3.1638	5.0713	9250	3.2396	605996800
3.2417	5.2083	9500	3.2346	622380800
3.1531	5.3454	9750	3.2283	638764800
3.1282	5.4825	10000	3.2207	655148800
3.1652	5.6195	10250	3.2134	671532800
3.2138	5.7566	10500	3.2076	687916800
3.1726	5.8936	10750	3.1993	704300800
3.1246	6.0307	11000	3.1959	720642560
3.0724	6.1678	11250	3.1955	737026560
3.0867	6.3048	11500	3.1913	753410560
3.1188	6.4419	11750	3.1831	769794560
3.0693	6.5789	12000	3.1795	786178560
3.1578	6.7160	12250	3.1704	802562560
3.118	6.8531	12500	3.1671	818946560
3.1048	6.9901	12750	3.1645	835330560
3.0561	7.1272	13000	3.1645	851672320
3.0638	7.2643	13250	3.1635	868056320
3.0684	7.4013	13500	3.1579	884440320
3.0546	7.5384	13750	3.1544	900824320
3.1225	7.6754	14000	3.1499	917208320
3.0423	7.8125	14250	3.1457	933592320
3.0468	7.9496	14500	3.1437	949976320
2.9983	8.0866	14750	3.1487	966318080
3.0739	8.2237	15000	3.1463	982702080
3.0517	8.3607	15250	3.1422	999086080
3.0804	8.4978	15500	3.1376	1015470080
3.0781	8.6349	15750	3.1330	1031854080
3.0242	8.7719	16000	3.1287	1048238080
3.0461	8.9090	16250	3.1266	1064622080
3.014	9.0461	16500	3.1325	1080963840
2.9533	9.1831	16750	3.1359	1097347840
2.9552	9.3202	17000	3.1306	1113731840
3.0019	9.4572	17250	3.1248	1130115840
3.0367	9.5943	17500	3.1240	1146499840
3.0225	9.7314	17750	3.1178	1162883840
3.0146	9.8684	18000	3.1173	1179267840
2.9585	10.0055	18250	3.1242	1195609600
2.9445	10.1425	18500	3.1262	1211993600
3.0177	10.2796	18750	3.1238	1228377600
2.9406	10.4167	19000	3.1203	1244761600
2.9815	10.5537	19250	3.1138	1261145600
2.9711	10.6908	19500	3.1097	1277529600
3.0437	10.8279	19750	3.1064	1293913600
3.0062	10.9649	20000	3.1035	1310297600
2.8808	11.1020	20250	3.1168	1326639360
2.9058	11.2390	20500	3.1165	1343023360
2.9946	11.3761	20750	3.1120	1359407360
2.9773	11.5132	21000	3.1091	1375791360
2.9939	11.6502	21250	3.1057	1392175360
3.0454	11.7873	21500	3.1038	1408559360
2.9825	11.9243	21750	3.0995	1424943360
2.8964	12.0614	22000	3.1141	1441285120
2.9369	12.1985	22250	3.1134	1457669120
2.9761	12.3355	22500	3.1094	1474053120
2.9143	12.4726	22750	3.1043	1490437120
2.9548	12.6096	23000	3.1032	1506821120
2.9505	12.7467	23250	3.0978	1523205120
2.9793	12.8838	23500	3.0932	1539589120
2.9114	13.0208	23750	3.1067	1555930880
2.895	13.1579	24000	3.1078	1572314880
2.9366	13.2950	24250	3.1055	1588698880
2.9259	13.4320	24500	3.1031	1605082880
3.0064	13.5691	24750	3.0971	1621466880
2.9068	13.7061	25000	3.0956	1637850880
2.9737	13.8432	25250	3.0901	1654234880
2.9672	13.9803	25500	3.0899	1670618880
2.943	14.1173	25750	3.1068	1686960640
2.9177	14.2544	26000	3.1012	1703344640
2.9234	14.3914	26250	3.0999	1719728640
2.9529	14.5285	26500	3.0934	1736112640
2.9084	14.6656	26750	3.0929	1752496640
2.9398	14.8026	27000	3.0884	1768880640
2.9637	14.9397	27250	3.0839	1785264640
2.9473	15.0768	27500	3.1024	1801606400
2.9212	15.2138	27750	3.1028	1817990400
2.9504	15.3509	28000	3.0967	1834374400
2.918	15.4879	28250	3.0943	1850758400
2.9953	15.625	28500	3.0878	1867142400
2.9286	15.7621	28750	3.0854	1883526400
2.9965	15.8991	29000	3.0812	1899910400
2.8275	16.0362	29250	3.0998	1916252160
2.9254	16.1732	29500	3.1027	1932636160
2.8747	16.3103	29750	3.0958	1949020160
3.0032	16.4474	30000	3.0933	1965404160
2.9734	16.5844	30250	3.0875	1981788160
2.9778	16.7215	30500	3.0822	1998172160
2.918	16.8586	30750	3.0825	2014556160
2.9584	16.9956	31000	3.0783	2030940160
2.9237	17.1327	31250	3.1016	2047281920
2.8865	17.2697	31500	3.0987	2063665920
2.8993	17.4068	31750	3.0901	2080049920
2.978	17.5439	32000	3.0852	2096433920
2.953	17.6809	32250	3.0789	2112817920
2.9651	17.8180	32500	3.0714	2129201920
2.8652	17.9550	32750	3.0586	2145585920
2.8281	18.0921	33000	3.0705	2161927680
2.8489	18.2292	33250	3.0678	2178311680
2.8807	18.3662	33500	3.0560	2194695680
2.8116	18.5033	33750	3.0484	2211079680
2.8966	18.6404	34000	3.0372	2227463680
2.822	18.7774	34250	3.0326	2243847680
2.8318	18.9145	34500	3.0223	2260231680
2.7053	19.0515	34750	3.0379	2276573440
2.7505	19.1886	35000	3.0334	2292957440
2.7068	19.3257	35250	3.0275	2309341440
2.733	19.4627	35500	3.0200	2325725440
2.6417	19.5998	35750	3.0145	2342109440
2.6484	19.7368	36000	3.0085	2358493440
2.7077	19.8739	36250	3.0017	2374877440

Framework versions

—Transformers 4.49.0
—Pytorch 2.5.1+cu121
—Datasets 4.8.5
—Tokenizers 0.21.4