Modelpublic

wriindonesia/seallm-7b-finetuned-small

sourceHugging Faceapache-2.0updated 11mo agoView on Hugging Face

0likes109downloads

Model Card

🌊 SeaLLM-7B-v2.5 — LoRA Adapter (EN→ID, Small-Sample-2, ≈1 K samples)

This repository provides a LoRA fine-tuned adapter for SeaLLMs/SeaLLM-7B-v2.5, trained on about 1,000 English–Indonesian translation pairs. It is part of WRI Indonesia’s exploration of efficient domain adaptation for scientific and environmental text translation using QLoRA.

🧠 Model Details

Property	Value
Base Model	`SeaLLMs/SeaLLM-7B-v2.5`
Adapter Type	LoRA (via PEFT)
Training Method	QLoRA (4-bit quantized fine-tuning)
Context Length	896 tokens
Training Size	≈ 1 000 parallel examples
Domain	Environment, forestry, climate policy
Compute	Single GPU (NVIDIA L40)
Training Time	~30 minutes
Epochs	3
Experiment Name	`small-sample-2`
Frameworks	🤗 Transformers / PEFT / Datasets

⚙️ Training Configuration

yaml

peft_type: LORA
r: 32
lora_alpha: 16
lora_dropout: 0.05
target_modules:
  - q_proj
  - v_proj
  - k_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
task_type: CAUSAL_LM
max_seq_length: 896
num_train_epochs: 3
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 2e-4
scheduler: cosine
warmup_ratio: 0.03
bf16: true
save_strategy: epoch
evaluation_strategy: epoch
early_stopping_patience: 2
load_best_model_at_end: true