Modelpublic

ymoslem/whisper-medium-ga2en-v6.3.2-15k-r

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

1likes10downloads

Model Card

Whisper Medium GA-EN Speech Translation

This model is a fine-tuned version of openai/whisper-medium on the IWSLT-2023, FLEURS, BiteSize, SpokenWords, Tatoeba, Wikimedia, and EUbookshop dataset. It achieves the following results on the evaluation set:

—Loss: 1.2038
—Bleu: 34.85
—Chrf: 54.43
—Wer: 60.9185

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.0001
—trainbatchsize: 16
—evalbatchsize: 16
—seed: 42
—optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
—lrschedulertype: linear
—lrschedulerwarmup_ratio: 0.03
—training_steps: 15000
—mixedprecisiontraining: Native AMP

Training results

Training Loss	Epoch	Step	Bleu	Chrf	Validation Loss	Wer
2.5219	0.0138	100	0.44	10.48	2.1106	107.2490
2.4608	0.0276	200	3.3	20.43	2.1816	179.1535
2.3008	0.0414	300	3.66	21.59	2.0587	206.4836
2.2095	0.0552	400	8.79	27.66	1.9459	100.3602
2.0454	0.0690	500	8.14	27.36	1.8681	122.1522
1.9937	0.0828	600	11.05	30.26	1.8717	97.2535
1.868	0.0966	700	9.14	29.03	1.7917	129.0410
1.9924	0.1103	800	12.62	33.2	1.7170	89.6443
1.8646	0.1241	900	11.98	30.77	1.7252	97.8838
1.7644	0.1379	1000	10.87	31.0	1.6832	109.1851
1.692	0.1517	1100	13.05	34.46	1.6837	93.3814
1.7044	0.1655	1200	20.95	37.42	1.5527	75.2364
1.6824	0.1793	1300	14.91	35.56	1.5611	92.6159
1.6557	0.1931	1400	14.0	36.54	1.5554	99.8199
1.5456	0.2069	1500	19.72	39.81	1.5058	83.5660
1.3755	0.2207	1600	18.04	37.95	1.5039	82.9806
1.3959	0.2345	1700	17.01	39.5	1.4374	85.2319
1.5012	0.2483	1800	14.93	39.24	1.4242	114.4079
1.4278	0.2621	1900	23.85	42.69	1.3904	73.0302
1.3285	0.2759	2000	17.7	37.23	1.4493	83.8811
1.2655	0.2897	2100	20.1	40.32	1.3661	79.7839
1.2074	0.3034	2200	24.45	43.79	1.3387	72.9851
1.1893	0.3172	2300	21.45	42.61	1.3308	82.3953
1.1236	0.3310	2400	22.77	44.17	1.3050	77.3075
1.0934	0.3448	2500	25.54	46.32	1.2793	72.2647
1.06	0.3586	2600	28.27	47.32	1.2396	65.6911
1.0327	0.3724	2700	28.45	47.01	1.2577	67.3570
1.1623	0.3862	2800	24.54	47.43	1.2194	73.6155
1.0215	0.4	2900	27.4	49.6	1.2039	69.2481
0.9185	0.4138	3000	27.04	49.24	1.1724	67.8973
0.9003	0.4276	3100	31.08	50.11	1.1674	63.8001
0.9839	0.4414	3200	30.24	50.63	1.1580	64.5655
0.9396	0.4552	3300	30.79	51.72	1.1202	64.9257
0.9051	0.4690	3400	30.34	53.08	1.1180	66.4566
0.8621	0.4828	3500	33.3	53.86	1.1042	60.7834
0.8236	0.4966	3600	32.77	53.21	1.1070	62.0441
0.829	0.5103	3700	32.49	54.21	1.0771	62.5844
0.8375	0.5241	3800	32.27	53.98	1.0780	63.0797
0.8206	0.5379	3900	33.26	55.07	1.0615	61.6389
0.8059	0.5517	4000	33.24	55.16	1.0552	61.5038
0.9133	0.5655	4100	29.38	49.22	1.2218	66.0964
1.051	0.5793	4200	25.12	46.01	1.2304	71.8145
0.954	0.5931	4300	25.47	45.88	1.2501	75.3715
0.939	0.6069	4400	29.19	47.63	1.2204	66.9068
0.9887	0.6207	4500	27.99	47.01	1.2099	67.7172
1.0044	0.6345	4600	23.77	45.33	1.2080	73.3904
0.9881	0.6483	4700	26.46	47.36	1.2188	68.5277
0.9674	0.6621	4800	26.11	45.92	1.2296	68.3026
0.8845	0.6759	4900	27.3	46.08	1.2347	68.0324
0.8297	0.6897	5000	29.48	48.96	1.2108	64.6105
0.9065	0.7034	5100	29.81	49.94	1.1873	64.2503
0.8096	0.7172	5200	28.5	46.93	1.2122	66.2314
0.8077	0.7310	5300	29.26	48.21	1.1945	64.4755
0.8227	0.7448	5400	26.82	48.43	1.2310	71.4093
0.7587	0.7586	5500	29.45	49.03	1.2067	65.3309
0.7206	0.7724	5600	29.89	49.33	1.2114	65.5561
0.8088	0.7862	5700	31.88	51.4	1.1689	64.2954
0.693	0.8	5800	27.23	48.11	1.1644	68.7078
0.7099	0.8138	5900	31.01	49.42	1.1852	63.3949
0.7564	0.8276	6000	28.3	50.34	1.1554	71.0941
0.584	0.8414	6100	34.79	51.69	1.1566	59.0725
0.6817	0.8552	6200	34.08	51.95	1.1245	59.8829
0.5968	0.8690	6300	32.4	51.59	1.1475	62.9896
0.6092	0.8828	6400	32.83	52.82	1.1250	62.5844
0.6325	0.8966	6500	29.29	51.68	1.1108	69.1130
0.6002	0.9103	6600	27.64	52.7	1.0993	71.0941
0.6247	0.9241	6700	28.39	52.4	1.0898	68.3026
0.6257	0.9379	6800	28.54	52.33	1.0863	70.9140
0.6719	0.9517	6900	31.43	53.53	1.0891	66.1414
0.4994	0.9655	7000	33.81	52.77	1.1066	61.0986
0.5469	0.9793	7100	30.52	53.13	1.0891	67.3570
0.6031	0.9931	7200	33.16	54.03	1.0933	62.1792
0.2469	1.0069	7300	33.76	52.38	1.1426	62.8546
0.2572	1.0207	7400	33.16	51.71	1.1292	64.8807
0.2762	1.0345	7500	34.76	54.28	1.1090	60.7384
0.2332	1.0483	7600	30.95	52.28	1.1073	66.1864
0.2069	1.0621	7700	32.39	53.08	1.0999	65.5561
0.2417	1.0759	7800	31.3	53.87	1.1008	65.1058
0.2403	1.0897	7900	32.18	53.3	1.1053	66.4566
0.208	1.1034	8000	32.0	52.48	1.1067	66.7717
0.3328	1.1172	8100	28.92	49.12	1.2137	68.4376
0.4045	1.1310	8200	28.47	51.53	1.2165	68.3926
0.4175	1.1448	8300	26.88	47.57	1.2790	74.5160
0.3976	1.1586	8400	21.56	44.64	1.3060	84.1513
0.4026	1.1724	8500	25.22	47.73	1.2476	73.1202
0.4088	1.1862	8600	26.03	48.08	1.2387	72.8050
0.4245	1.2	8700	29.8	49.69	1.2136	67.4021
0.4083	1.2138	8800	26.26	48.23	1.2784	73.4804
0.3832	1.2276	8900	29.06	49.36	1.2527	66.4115
0.4335	1.2414	9000	30.11	49.24	1.2772	67.2670
0.4056	1.2552	9100	32.51	50.18	1.3013	63.3048
0.3877	1.2690	9200	26.91	47.47	1.2897	71.5894
0.3787	1.2828	9300	1.2430	30.16	50.61	65.1058
0.3947	1.2966	9400	1.2318	29.9	50.77	66.0964
0.3908	1.3103	9500	1.1927	30.7	51.62	64.6105
0.405	1.3241	9600	1.2249	26.56	49.05	71.7695
0.3847	1.3379	9700	1.2105	33.22	51.98	61.8640
0.3674	1.3517	9800	1.2545	30.93	50.34	65.6011
0.3642	1.3655	9900	1.2443	25.23	47.97	77.9379
0.3636	1.3793	10000	1.2796	26.78	48.07	73.6155
0.329	1.3931	10100	1.2373	29.06	49.55	66.4566
0.4195	1.4069	10200	1.2187	29.11	50.65	66.2314
0.4244	1.4207	10300	1.2346	27.97	49.86	69.0680
0.3338	1.4345	10400	1.2239	29.96	50.45	66.0063
0.3401	1.4483	10500	1.2501	29.84	51.0	65.6911
0.3792	1.4621	10600	1.2353	28.38	49.19	69.1130
0.3549	1.4759	10700	1.2178	28.63	49.73	68.5727
0.3326	1.4897	10800	1.1936	29.57	51.1	64.4755
0.3418	1.5034	10900	1.1741	33.06	52.86	60.9185
0.3143	1.5172	11000	1.2046	31.49	50.4	63.5750
0.3245	1.5310	11100	1.2145	30.9	50.17	64.6105
0.3268	1.5448	11200	1.2119	33.5	53.0	60.2431
0.2894	1.5586	11300	1.2126	32.01	52.17	61.0986
0.2702	1.5724	11400	1.2213	31.33	50.89	63.7551
0.2876	1.5862	11500	1.2126	31.44	51.28	63.1697
0.2759	1.6	11600	1.2283	30.49	51.02	64.7456
0.2902	1.6138	11700	1.2205	32.33	50.53	63.2148
0.2638	1.6276	11800	1.2097	31.89	51.14	62.6745
0.2605	1.6414	11900	1.2129	31.35	50.63	63.3048
0.2374	1.6552	12000	1.2319	31.48	51.73	63.4849
0.2436	1.6690	12100	1.2219	30.43	50.92	65.5110
0.2366	1.6828	12200	1.2367	31.64	51.14	64.7006
0.218	1.6966	12300	1.2142	30.8	51.63	64.1153
0.2313	1.7103	12400	1.1877	30.8	50.63	64.5655
0.2307	1.7241	12500	1.1817	32.22	51.41	63.3498
0.2638	1.7379	12600	1.1514	33.74	52.11	60.6033
0.2211	1.7517	12700	1.1563	30.71	52.07	64.5655
0.197	1.7655	12800	1.1941	32.22	52.9	62.8546
0.2307	1.7793	12900	1.1771	32.83	52.96	62.7645
0.198	1.7931	13000	1.1908	32.16	51.85	63.9352
0.1716	1.8069	13100	1.2065	31.91	51.37	62.6294
0.2031	1.8207	13200	1.1745	31.83	51.86	64.0252
0.1785	1.8345	13300	1.1607	31.33	52.57	64.7006
0.2013	1.8483	13400	1.1785	33.29	53.34	62.6745
0.1842	1.8621	13500	1.1723	34.41	54.31	60.0630
0.2015	1.8759	13600	1.1859	32.88	53.07	62.2692
0.1848	1.8897	13700	1.1668	33.62	53.75	62.8095
0.1394	1.9034	13800	1.1734	34.33	54.03	61.2787
0.1774	1.9172	13900	1.1735	32.63	53.37	62.8996
0.1506	1.9310	14000	1.1768	35.17	54.34	59.4327
0.1399	1.9448	14100	1.1827	33.68	53.8	62.1792
0.1434	1.9586	14200	1.1721	34.62	54.24	60.9185
0.1203	1.9724	14300	1.1733	34.08	53.75	61.8190
0.1417	1.9862	14400	1.1615	33.98	54.19	62.1792
0.1458	2.0	14500	1.1739	33.65	53.31	62.9896
0.07	2.0138	14600	1.1916	33.98	53.96	61.9090
0.051	2.0276	14700	1.1967	34.13	54.36	61.1887
0.0481	2.0414	14800	1.2024	34.06	54.38	61.4588
0.0574	2.0552	14900	1.2038	34.23	54.08	61.2787
0.0621	2.0690	15000	1.2038	34.85	54.43	60.9185

Framework versions

—Transformers 4.41.2
—Pytorch 2.2.0+cu121
—Datasets 2.20.0
—Tokenizers 0.19.1

Citation

@inproceedings{moslem-2024-leveraging,
    title = "Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation",
    author = "Moslem, Yasmin",
    booktitle = "Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand (in-person and online)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.iwslt-1.31/",
    doi = "10.18653/v1/2024.iwslt-1.31",
    pages = "265--273",
    abstract = "This paper describes our system submission to the International Conference on Spoken Language Translation (IWSLT 2024) for Irish-to-English speech translation. We built end-to-end systems based on Whisper, and employed a number of data augmentation techniques, such as speech back-translation and noise augmentation. We investigate the effect of using synthetic audio data and discuss several methods for enriching signal diversity."
}