Modelpublic

Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-sni-switch-top1-router-ffn-lora-aux-anneal-5ep

sourceHugging Faceupdated 11d agoView on Hugging Face

0likes286downloads

Model Card

llama-3.1-8b-instruct-4x1-moe-sni-switch-top1-router-ffn-lora-aux-anneal-5ep

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

—Loss: 0.3883
—Router Entropy: 1.1134
—Router Max Load: 0.2685
—Expert 0 Utilization: 0.2470
—Expert 1 Utilization: 0.2418
—Expert 2 Utilization: 0.2427
—Expert 3 Utilization: 0.2685
—Router Aux Loss: 0.9996
—Router Aux Loss Coef: 0.0010
—Router Aux Loss Weighted: 0.0010
—Router Z Loss: 0.4890

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 2e-05
—trainbatchsize: 1
—evalbatchsize: 1
—seed: 42
—distributed_type: multi-GPU
—num_devices: 2
—gradientaccumulationsteps: 16
—totaltrainbatch_size: 32
—totalevalbatch_size: 2
—optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
—lrschedulertype: linear
—lrschedulerwarmup_steps: 0.03
—num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss	Entropy	Max Load	0 Utilization	1 Utilization	2 Utilization	3 Utilization	Aux Loss	Aux Loss Coef	Aux Loss Weighted	Z Loss
1.2114	0.0920	200	0.8261	1.2630	0.3386	0.2036	0.2084	0.2494	0.3386	1.0212	0.0100	0.0102	2.2378
0.8468	0.1841	400	0.5183	1.1628	0.3241	0.2220	0.2014	0.2525	0.3241	1.0096	0.0100	0.0101	2.1002
0.7673	0.2761	600	0.4561	1.1255	0.2858	0.2390	0.2324	0.2428	0.2858	0.9988	0.0100	0.0100	1.7220
0.5786	0.3681	800	0.4324	1.0956	0.2785	0.2421	0.2356	0.2438	0.2785	0.9982	0.0100	0.0100	1.4931
0.7534	0.4602	1000	0.4127	1.0907	0.2770	0.2411	0.2361	0.2458	0.2770	0.9983	0.0100	0.0100	1.3975
0.6043	0.5522	1200	0.3988	1.0752	0.2765	0.2466	0.2321	0.2448	0.2765	0.9982	0.0100	0.0100	1.2850
0.6419	0.6442	1400	0.3935	1.0829	0.2723	0.2519	0.2305	0.2453	0.2723	0.9978	0.0100	0.0100	1.1872
0.6211	0.7362	1600	0.3857	1.0759	0.2727	0.2509	0.2292	0.2472	0.2727	0.9983	0.0100	0.0100	1.1512
0.6462	0.8283	1800	0.3787	1.0726	0.2708	0.2460	0.2354	0.2479	0.2708	0.9978	0.0100	0.0100	1.0753
0.5454	0.9203	2000	0.3734	1.0710	0.2733	0.2572	0.2227	0.2468	0.2733	0.9987	0.0100	0.0100	1.0610
0.5873	1.0120	2200	0.3709	1.0572	0.2676	0.2506	0.2371	0.2446	0.2676	0.9979	0.0100	0.0100	1.0066
0.5866	1.1040	2400	0.3678	1.0561	0.2634	0.2506	0.2340	0.2520	0.2634	0.9980	0.0098	0.0097	0.9194
0.6267	1.1960	2600	0.3640	1.0744	0.2674	0.2560	0.2340	0.2426	0.2674	0.9989	0.0096	0.0095	0.8921
0.5447	1.2881	2800	0.3631	1.0794	0.2684	0.2557	0.2303	0.2456	0.2684	0.9988	0.0094	0.0093	0.8736
0.6235	1.3801	3000	0.3598	1.0811	0.2634	0.2547	0.2363	0.2457	0.2634	0.9989	0.0091	0.0091	0.8421
0.5394	1.4721	3200	0.3574	1.0903	0.2608	0.2545	0.2344	0.2503	0.2608	0.9987	0.0089	0.0089	0.8101
0.4995	1.5641	3400	0.3554	1.0742	0.2666	0.2513	0.2368	0.2453	0.2666	0.9987	0.0087	0.0087	0.8077
0.5477	1.6562	3600	0.3537	1.0793	0.2651	0.2532	0.2340	0.2477	0.2651	0.9988	0.0085	0.0085	0.7793
0.5979	1.7482	3800	0.3515	1.0877	0.2694	0.2508	0.2331	0.2467	0.2694	0.9988	0.0083	0.0083	0.7670
0.5900	1.8402	4000	0.3512	1.0840	0.2654	0.2508	0.2368	0.2471	0.2654	0.9990	0.0081	0.0081	0.7494
0.4870	1.9323	4200	0.3485	1.0928	0.2620	0.2530	0.2384	0.2466	0.2620	0.9990	0.0079	0.0079	0.7250
0.4668	2.0239	4400	0.3581	1.0786	0.2670	0.2503	0.2355	0.2472	0.2670	0.9990	0.0077	0.0077	0.7038
0.6377	2.1160	4600	0.3574	1.0814	0.2636	0.2528	0.2378	0.2458	0.2636	0.9987	0.0075	0.0075	0.6586
0.4402	2.2080	4800	0.3574	1.1028	0.2698	0.2445	0.2424	0.2433	0.2698	0.9989	0.0073	0.0073	0.6282
0.4786	2.3000	5000	0.3573	1.1062	0.2684	0.2486	0.2373	0.2457	0.2684	0.9991	0.0071	0.0071	0.6280
0.3695	2.3921	5200	0.3585	1.0941	0.2693	0.2487	0.2389	0.2431	0.2693	0.9992	0.0069	0.0069	0.6454
0.5038	2.4841	5400	0.3570	1.1056	0.2634	0.2494	0.2402	0.2471	0.2634	0.9992	0.0067	0.0067	0.6188
0.4915	2.5761	5600	0.3561	1.1073	0.2714	0.2494	0.2370	0.2422	0.2714	0.9998	0.0065	0.0065	0.6281
0.4836	2.6681	5800	0.3543	1.1026	0.2715	0.2477	0.2377	0.2431	0.2715	0.9995	0.0062	0.0062	0.6319
0.4521	2.7602	6000	0.3536	1.1027	0.2665	0.2477	0.2433	0.2424	0.2665	0.9994	0.0060	0.0060	0.6062
0.4057	2.8522	6200	0.3540	1.1056	0.2697	0.2499	0.2355	0.2450	0.2697	0.9997	0.0058	0.0058	0.6074
0.4003	2.9442	6400	0.3519	1.1136	0.2649	0.2474	0.2363	0.2515	0.2649	0.9993	0.0056	0.0056	0.5839
0.3540	3.0359	6600	0.3713	1.1024	0.2597	0.2475	0.2445	0.2484	0.2597	0.9995	0.0054	0.0054	0.5655
0.4863	3.1279	6800	0.3705	1.1005	0.2640	0.2479	0.2416	0.2466	0.2640	0.9994	0.0052	0.0052	0.5580
0.3915	3.2200	7000	0.3745	1.1044	0.2647	0.2513	0.2356	0.2484	0.2647	0.9995	0.0050	0.0050	0.5579
0.3265	3.3120	7200	0.3678	1.1041	0.2697	0.2481	0.2403	0.2419	0.2697	0.9998	0.0048	0.0048	0.5539
0.3505	3.4040	7400	0.3691	1.1099	0.2693	0.2472	0.2455	0.2380	0.2693	0.9997	0.0046	0.0046	0.5456
0.4006	3.4960	7600	0.3718	1.0990	0.2693	0.2455	0.2412	0.2441	0.2693	0.9994	0.0044	0.0044	0.5454
0.4109	3.5881	7800	0.3703	1.1028	0.2675	0.2509	0.2374	0.2442	0.2675	0.9996	0.0042	0.0042	0.5401
0.3907	3.6801	8000	0.3711	1.1062	0.2661	0.2479	0.2428	0.2432	0.2661	0.9996	0.0040	0.0040	0.5373
0.4161	3.7721	8200	0.3655	1.1078	0.2656	0.2445	0.2466	0.2433	0.2656	0.9994	0.0038	0.0038	0.5341
0.3312	3.8642	8400	0.3671	1.1109	0.2654	0.2472	0.2418	0.2456	0.2654	0.9995	0.0036	0.0036	0.5332
0.4079	3.9562	8600	0.3666	1.1115	0.2676	0.2472	0.2398	0.2453	0.2676	0.9995	0.0033	0.0033	0.5256
0.3204	4.0479	8800	0.3853	1.1129	0.2620	0.2466	0.2441	0.2474	0.2620	0.9996	0.0031	0.0031	0.5082
0.2505	4.1399	9000	0.3903	1.1091	0.2658	0.2467	0.2433	0.2442	0.2658	0.9996	0.0029	0.0029	0.5080
0.4093	4.2319	9200	0.3895	1.1099	0.2670	0.2463	0.2457	0.2409	0.2670	0.9997	0.0027	0.0027	0.5042
0.3267	4.3239	9400	0.3872	1.1165	0.2674	0.2494	0.2403	0.2429	0.2674	0.9997	0.0025	0.0025	0.4896
0.3335	4.4160	9600	0.3943	1.1158	0.2642	0.2471	0.2421	0.2466	0.2642	0.9996	0.0023	0.0023	0.4902
0.3007	4.5080	9800	0.3910	1.1092	0.2674	0.2463	0.2411	0.2452	0.2674	0.9995	0.0021	0.0021	0.4944
0.2344	4.6000	10000	0.3907	1.1162	0.2684	0.2489	0.2389	0.2438	0.2684	0.9997	0.0019	0.0019	0.4887
0.2929	4.6921	10200	0.3881	1.1125	0.2693	0.2453	0.2416	0.2438	0.2693	0.9997	0.0017	0.0017	0.4992
0.3171	4.7841	10400	0.3877	1.1109	0.2688	0.2463	0.2407	0.2442	0.2688	0.9996	0.0015	0.0015	0.4946
0.3340	4.8761	10600	0.3894	1.1144	0.2680	0.2477	0.2407	0.2436	0.2680	0.9995	0.0013	0.0013	0.4870
0.2223	4.9682	10800	0.3884	1.1133	0.2715	0.2471	0.2417	0.2397	0.2715	0.9997	0.0011	0.0011	0.4893
0.2070	5.0	10870	0.3883	1.1134	0.2685	0.2470	0.2418	0.2427	0.2685	0.9996	0.0010	0.0010	0.4890

Framework versions

—PEFT 0.19.1
—Transformers 5.9.0
—Pytorch 2.11.0+cu128
—Datasets 4.8.5
—Tokenizers 0.22.2