RobbedoesHF/geitje-ultra-dutch-definition-expansion-qlora

QLoRA Adapter for Dutch Definition Expansion (GEITje-7B-ultra)

This repository contains a QLoRA adapter fine-tuned on BramVanroy/GEITje-7B-ultra for the task of sense-preserving definitional expansion in Dutch.

This work was developed as part of the Master's thesis, ["Transformer-based Expansion of Dutch Dictionary Definitions"](https://github.com/RobbeDoesPy/Dutch-Dictionary-Definitions-Expansion/blob/main/Transformer-based_Expansion_of_Dutch_Dictionary_Definitions.pdf), submitted for the degree of Master of Science in Artificial Intelligence at KU Leuven.

About the Thesis

The research investigates the potential of transformer-based models to automate a significant bottleneck in contemporary lexicography: the manual expansion of concise, core-meaning definitions into comprehensive, formally structured dictionary entries. The study focuses on Dutch, a task requiring not only semantic accuracy but also strict adherence to lexicographical style and structure.

The thesis empirically compares two primary methodologies: in-context learning via few-shot prompting and adaptation via parameter-efficient fine-tuning (specifically, QLoRA). This comparison was conducted across a range of powerful multilingual and Dutch-specific models, including mT5-xl, GEITje Ultra, Aya-101, and Aya-23, to determine the most effective strategy for this high-precision domain.

This Model's Role and Performance

This fine-tuned GEITje-7B-ultra model was a key subject in the study for testing the domain mismatch hypothesis. As a model heavily aligned for conversational interaction its performance was analysed to see if this stylistic prior would conflict with the formal structured nature of lexicographical text.

After fine-tuning GEITje Ultra emerged as a top-performing model distinguished by its strong performance on lexical overlap metrics suggesting it was highly effective at learning the precise phrasing of the training data. While the fine-tuned Aya-23 model achieved slightly higher scores on semantic quality metrics, GEITje Ultra proved to be a highly competitive model demonstrating that task-specific fine-tuning can successfully adapt a conversationally-aligned model to a formal domain.

How to Use

To use this adapter you must first load the base model (`BramVanroy/GEITje-7B-ultra') in 4-bit and then apply this adapter on top of it.

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model_id = "BramVanroy/GEITje-7B-ultra"
adapter_id = "RobbedoesHF/geitje-7b-ultra-dutch-definition-expansion-qlora" # The repo ID of this adapter

# Load the base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2", # Recommended for GEITje
)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
tokenizer.pad_token = tokenizer.eos_token # Set pad token

# Apply the LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)

print("Model loaded successfully!")

Prompting Format

This adapter was fine-tuned on a specific instructional prompt. For best results, your input should match this structure.

# Define the lemma and short definition you want to expand
lemma = "ecoroman"
short_def = "roman over milieuproblematiek"


# Create the chat prompt using the tokenizer's template
chat = [
    {"role": "system", "content": "Je bent een expert-lexicograaf die definities schrijft voor een Nederlands woordenboek."},
    {"role": "user", "content": f"Breid de volgende korte definitie voor het woord '{lemma}' uit tot een volledige definitie: '{short_def}'"}
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

# Tokenize the prompt
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# Generate the output tokens
print("
Generating definition...")
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512, # Chosen based on the longest full definition's token length for this model
        num_beams=4, # What was used for the thesis
        early_stopping=True,
        pad_token_id=tokenizer.eos_token_id
    )

# Decode and clean the output
# The output includes the prompt so we split for the assistant's response
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
assistant_response = decoded_output.split("<|assistant|>")[1].strip()

print("\n--- Prompt ---")
print(prompt)
print("\n--- Model Output ---")
print(decoded_output)