Modelpublic

mesolitica/Malaysian-Qwen2.5-14B-Instruct-FP8

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes22downloads

Model Card

Malaysian Qwen 2.5 14B Instruct Dynamic FP8

This is FP8 Dynamic Quantization (A8W8) for https://huggingface.co/mesolitica/Malaysian-Qwen2.5-14B-Instruct

Benchmark

MalayMMLU

Based on 0-shot exact first token match using vLLM,

                                        Model   Accuracy  shot        category
0  Malaysian-Qwen2.5-14B-Instruct-FP8-Dynamic  72.329104     0            STEM
1  Malaysian-Qwen2.5-14B-Instruct-FP8-Dynamic  72.026081     0        Language
2  Malaysian-Qwen2.5-14B-Instruct-FP8-Dynamic  66.088465     0  Social science
3  Malaysian-Qwen2.5-14B-Instruct-FP8-Dynamic  66.274886     0          Others
4  Malaysian-Qwen2.5-14B-Instruct-FP8-Dynamic  70.716724     0      Humanities
Model : Malaysian-Qwen2.5-14B-Instruct-FP8-Dynamic
Metric : full
Shot : 0
average accuracy 69.13228431008136
accuracy for STEM 72.32910356119525
accuracy for Language 72.02608142493638
accuracy for Social science 66.08846487424111
accuracy for Others 66.27488606380427
accuracy for Humanities 70.71672354948805

Acknowledgement

Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!