ikhou/qwen35-2b-qlora-bilingual-ckpt8000

Qwen3.5-2B QLoRA Bilingual Adapter

This repository contains the selected best saved LoRA adapter checkpoint from an in-progress Ikhou supervised fine-tuning run on Qwen/Qwen3.5-2B.

The original run was configured for a very long wall-clock duration, so the checkpoint was selected early from the saved checkpoints based on validation loss instead of waiting for the full run to complete.

Selected Checkpoint

• —Selected checkpoint: checkpoint-8000
• —Base model: Qwen/Qwen3.5-2B
• —Training method: QLoRA 4-bit
• —Precision during training: bf16 compute
• —LoRA rank / alpha / dropout: r=16, alpha=32, dropout=0.05
• —Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Why This Checkpoint

Among the checkpoints still available locally, checkpoint-8000 had the best validation loss.

• —checkpoint-8000: eval_loss = 0.09412
• —checkpoint-8500: eval_loss = 0.10018
• —checkpoint-9000: eval_loss = 0.09798

Earlier checkpoints achieved slightly lower validation loss during the same run, but they were no longer retained locally because the training job used a limited checkpoint retention policy.

Intended Use

This adapter is intended for short bilingual dictionary-style glosses and translation-style responses in the Ikhou workflow.

Because this is a PEFT adapter repo, you should load it on top of the base model rather than treat it as a full standalone model snapshot.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen3.5-2B"
adapter_id = "ikhou/qwen35-2b-qlora-bilingual-ckpt8000"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    trust_remote_code=True,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

If you need a merged standalone model, merge the adapter into the base model after loading.

Notes

• —This repo contains inference-time adapter artifacts only.
• —Optimizer state and trainer state were intentionally excluded from upload.
• —The source training job continued past this checkpoint, but later saved checkpoints did not improve on validation loss.