Modelpublic

JoshyG14/qwen3-1.7b-spanish-lora-with-inject

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes19downloads

Model Card

Qwen3-1.7B LoRA-with-inject

LoRA adapter for Qwen/Qwen3-1.7B, trained for Spanish sentence generation under morphosyntactic constraints with the gold inflected form supplied in the prompt. This is Experiment A (LoRA-with-inject) from the MSc project: Grammatically Controlled Sentence Generation with Small Language Models for Language Learning.

Code: github.com/JoshuaGraham14/LinguistOS.

Companion adapter trained without form injection: JoshyG14/qwen3-1.7b-spanish-lora-no-inject.

Intended use

Given a Spanish verb lemma, morphosyntactic constraints (tense, person), and the required surface form, generate a short natural sentence in which that form is the main verb.

The adapter is not a general-purpose Spanish chat model. Load it on top of Qwen/Qwen3-1.7B and use the same form-injection prompt format as at training time (see the repository configs).

Training


Base model	`Qwen/Qwen3-1.7B` (Apache 2.0)
Method	LoRA, rank \(r=16\), \(\alpha=32\), dropout 0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Data	4,205 filtered prompt–completion pairs from the 150-verb diagnostic grid (3,995 train / 210 val; hard cells oversampled \(\times 2\))
Objective	Supervised fine-tuning (next-token CE on prompt–completion pairs)
Checkpoint	Best validation loss at 1,000 steps (early stopping); 3 epochs, lr \(2\times10^{-4}\)
Adapter size	~67 MB

Load

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-1.7B"
adapter_id = "JoshyG14/qwen3-1.7b-spanish-lora-with-inject"

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(model, adapter_id)

Framework versions

PEFT 0.19.1 · TRL 1.8.0 · Transformers 5.13.0 · PyTorch 2.12.1