mohamedrayyan/gemma-e4b-dhivehi-lora-v1

Gemma E4B Dhivehi LoRA Pilot

This is a private pilot LoRA adapter for google/gemma-4-E4B-it, trained for early Dhivehi instruction-following experiments. It is not a production model.

Intended Scope

The adapter is intended for small experiments around:

• —Dhivehi chat in Thaana script.
• —Polite rewriting.
• —Grammar/spelling correction.
• —Short summarization.
• —Simple translation checks.

It should not be used for legal, medical, financial, emergency, official, or high-stakes advice.

Training Data

Training data is synthetic and stored in the private dataset repo mohamedrayyan/dhivehi-synthetic-v1.

Current adapter run used:

• —Train file: scale-combined-v1/train.jsonl
• —Held-out file: scale-combined-v1/heldout.jsonl
• —Train records: 64,679
• —Held-out records: 1,420
• —Approx train tokens: 10,039,638

scale-combined-v1 combines the earlier pilot split with the first larger controlled synthetic scale run. It includes Dhivehi rewrite, grammar correction, summary, chat, and English-to-Dhivehi translation rows.

Training

The current adapter was trained on Modal H100 with PEFT LoRA.

Important implementation detail: Gemma E4B is multimodal. LoRA target modules must be restricted to exact model.language_model modules. A previous smoke run accidentally targeted audio/vision tower internals; that adapter was superseded.

Corrected adapter properties:

• —LoRA target modules: 258 exact model.language_model linear modules.
• —Adapter parameters: 34,881,536.
• —Audio tower LoRA tensors: 0.
• —Vision tower LoRA tensors: 0.
• —Adapter size: about 140 MB.

Current 500-step scale-combined training:

Preserved tags:

• —quality-fixed-smoke-20260531: corrected quality-only adapter.
• —controlled-batch-1-sft-20260531: controlled-only adapter, useful for comparison but overfit to narrow templates.
• —pilot-combined-v1-sft-20260531: earlier combined pilot adapter.
• —scale-combined-v1-sft-20260602: current scale adapter.

Evaluation

Base-vs-adapter eval artifacts are uploaded in this model repo under:

• —eval/scale-combined-v1-sft-eval-1780422776/summary.json
• —eval/scale-combined-v1-sft-eval-1780422776/outputs.jsonl
• —eval/pilot-combined-v1-eval-smoke-1780254244/summary.json
• —eval/pilot-combined-v1-eval-smoke-1780254244/outputs.jsonl

Current scale fixed eval, 12 prompts:

Qualitative notes:

• —The adapter gives shorter, more Dhivehi-heavy outputs than the base model.
• —English-to-Dhivehi translation has more training support than the earlier pilot, but still needs native-speaker review.
• —Dhivehi-to-English translation can be correct on simple examples, but has very little training support.
• —Chat/rewrite outputs can repeat or preserve prompt wording too closely.
• —A separate 20-step DPO smoke run was completed in mohamedrayyan/gemma-e4b-dhivehi-dpo-v1, but it did not beat this SFT adapter on the current smoke eval.

Limitations

• —Synthetic training set with repeated local templates.
• —No native-speaker rating pass yet.
• —Translation quality is not stable.
• —The adapter may overfit to short school/homework-style rewriting and correction patterns.
• —The model may hallucinate details or produce mixed-script text.
• —This is a pilot adapter, not a general Dhivehi assistant.

Next Gate

Before running another DPO/ORPO pass:

• —Run a native-speaker review on samples from scale-combined-v1-sft-20260602.
• —Add broader local templates and longer source paragraphs before spending more data budget.
• —Separately curate translation and chat data.
• —Build at least 1K higher-quality preference pairs from real SFT failure cases.