PixelSpaceAI/Malaysian-Qwen2.5-7B-AWS-Malay-LoRA

Malaysian-Qwen2.5-7B-AWS-Malay-LoRA

A LoRA adapter that teaches `mesolitica/Malaysian-Qwen2.5-7B-Instruct` to answer AWS questions in a concise, natural Bahasa Melayu voice, keeping technical terms in English the way Malaysian engineers actually speak (S3, Lambda, IAM, bucket, policy).

What it does well (and what it doesn't)

This adapter changes how the model talks, not what it knows. That distinction is the whole story of this project.

• —✅ Voice / register / length. Base model answers AWS questions in long English-flavoured markdown; this adapter answers in tight, fluent Malay. This transferred cleanly from ~2.5k pairs.
• —✅ Well-covered topics (S3 tiers, Lambda basics, Route 53, DynamoDB) are fluent and mostly correct.
• —⚠️ Thin-coverage facts are unreliable. Topics seen only a handful of times in training (e.g. Lambda SnapStart, CloudWatch Logs Insights) can be stated confidently and wrongly. Fine-tuning at this data scale installs distribution, not individual facts.

For factual accuracy, pair this adapter with retrieval (RAG). In our tests, grounding generation on a retrieved AWS doc chunk fixed the fact errors while this adapter kept the Malay voice. Use the adapter for tone, use retrieval for truth.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "mesolitica/Malaysian-Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "PixelSpaceAI/Malaysian-Qwen2.5-7B-AWS-Malay-LoRA")

msgs = [{"role": "user", "content": "Apakah perbezaan antara Amazon S3 Standard dan S3 Glacier?"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=400, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Training

• —Base: mesolitica/Malaysian-Qwen2.5-7B-Instruct (Qwen chat template)
• —Data: ~2.5k AWS Q&A pairs in Bahasa Melayu (hand-authored serverless pairs + AWS official FAQs translated EN→MS), plus a small general-Malay blend to reduce forgetting. Held-out eval of 150 pairs.
• —Method: LoRA, r=16, alpha=32, dropout=0.05, target modules q,k,v,o,gate,up,down. lr 2e-4, cosine, batch 2 × grad-accum 4, max_length=1024, gradient checkpointing, assistant_only_loss=True.
• —Epochs: 3 with load_best_model_at_end; eval loss floors at epoch 2 (best checkpoint shipped).
• —Hardware: 1× RTX PRO 6000 Blackwell, ~20 GB peak VRAM (bf16 LoRA), ~15 min.

Limitations

• —Do not rely on this model for AWS facts without retrieval grounding.
• —Malay only; English or other languages fall back to base behaviour.
• —Knowledge is frozen at training time and does not track AWS service changes.

Adapter only (~161 MB). Requires the base model at inference.

Acknowledgements & attribution

This adapter builds directly on the work of others:

• —Base model: `mesolitica/Malaysian-Qwen2.5-7B-Instruct` by Mesolitica — a continue-finetune on ~1.5B tokens of curated Malaysian instruction data.
• —Upstream base: `Qwen/Qwen2.5-7B-Instruct` by Alibaba (Qwen team), Apache-2.0.

All credit for the underlying Malay language capability belongs to Mesolitica and the Qwen team; this repo only adds a small task-specific LoRA adapter on top.

License note: this repo is tagged Apache-2.0 by inheritance from Qwen2.5-7B-Instruct. Mesolitica's intermediate model does not state an explicit license on its model page; if you require licensing certainty for the base weights, confirm directly with Mesolitica.