mesolitica/Malaysian-Mistral-Small-3.1-24B-Instruct-2503

Malaysian Mistral-Small-3.1-24B-Instruct-2503

Continue finetuning https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 on highly curated 1.5B tokens Malaysian instruction dataset.

Improvement

1. 1.Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
2. 2.Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
3. 3.Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

1. 1.LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
2. 2.256 Rank with alpha 512, or alpha of 2.0
3. 3.Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
4. 4.Chunk CCE loss for LoRA.
5. 5.WanDB at https://wandb.ai/huseinzol05/lora-embedding-256-Mistral-Small-3.1-24B-Instruct-2503-malaysian-8k

Source code at https://github.com/mesolitica/malaya/tree/master/session/mistral3

Benchmark

MalayMMLU

Probability next tokens

Based on 0-shot official MalayMMLU First token accuracy,

While the original model,

First token match using vLLM

Based on 0-shot exact first token match using vLLM Guided Decoding,

While the original model,

Acknowledgement

Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!