DS4AI-UPB/gemma4-ro-re-lora

Gemma 4 31B QLoRA adapter — Romanian/English Relation Classification

Dragoș Mitruț Vasile · Elena-Simona Apostol · Stefan-Adrian Toma · Adrian Paschke · Ciprian-Octavian Truică

    

QLoRA adapter for google/gemma-4-31b-it, fine-tuned for Relation Classification on a Romanian translation of SemEval-2010 Task 8 plus the original English data. Given a sentence with two marked entities (<e1>, <e2>), the model assigns one of ten directional relations.

<!-- This adapter accompanies the SYNASC 2026 paper "Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian". -->

Results (macro F1-Score, SemEval-2010 Task 8 test set)

On classification, this 31B model is statistically indistinguishable from a fine-tuned XLM-RoBERTa-large (560M) on both languages (paired bootstrap, p = 0.23 English, p = 0.09 Romanian).

Training

• —Base: google/gemma-4-31b-it, loaded in 4-bit
• —LoRA: rank 32, alpha 64, dropout 0.05, all attention and MLP projections
• —3 epochs, effective batch size 16, peak LR 2e-4, cosine decay, 5% warmup
• —Combined English + Romanian training data (15,871 examples)
• —Single NVIDIA A100 40GB, about 5h13min

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("google/gemma-4-31b-it", load_in_4bit=True, device_map="auto")
model = PeftModel.from_pretrained(base, "<HF_REPO_RE>")
tok = AutoTokenizer.from_pretrained("<HF_REPO_RE>")

Prompt format and the full relation label set are in the run_inference.py script in the code repository.

Limitations

The Romanian training data is machine-translated with automatic post-validation, not a human gold standard. See the paper and the code repository for the translation quality analysis.

Citation

@misc{vasile2026crosslingual,
  title  = {Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian},
  author = {Vasile, Drago\c{s}-Mitru\c{t} and Apostol, Elena-Simona and Toma, \c{S}tefan-Adrian and Paschke, Adrian and Truic\u{a}, Ciprian-Octavian},
  year   = {2026},
  note   = {Preprint}
}