hannah-khallaf/e2r-strategy-gemma-3-12b-pairwise-qlora

E2R Strategy Gemma 3 12B Pairwise QLoRA

This repository contains the PEFT/QLoRA adapter for a Gemma 3 12B pairwise multilabel classifier for identifying Easy-to-Read (E2R) simplification strategies.

Base model

google/gemma-3-12b-it

Task

Given a standard sentence and its Easy-to-Read rewrite, the model scores each candidate strategy independently.

The six strategy labels, in benchmark order, are:

1. 1.Synonymy
2. 2.Modulation
3. 3.Compression
4. 4.Explanation
5. 5.Syntactic Change
6. 6.Omission

The model therefore performs six binary candidate-label evaluations for each sentence pair.

Model formulation

The checkpoint uses:

• —pair input: standard sentence + Easy-to-Read sentence;
• —pairwise binary-relevance formulation;
• —QLoRA;
• —all negative candidate labels during training;
• —token-probability decisions based on the next-token probabilities of true and false;
• —Gemma 3 12B Instruct as the base model.

Development-selected thresholds

The thresholds were selected using the development set only:

The thresholds are also provided in thresholds.json.

Recovered checkpoint

This repository contains seed 13 from a reproducibility rerun of the six-label benchmark.

This is not claimed to be the byte-identical historical checkpoint from the earlier experiment, because that checkpoint was no longer available. The model was retrained from the preserved experiment configuration.

Development performance

• —Macro-F1: 0.713499
• —Micro-F1: 0.773989
• —Exact-set accuracy: 0.335878
• —Hamming loss: 0.181298

Test performance

• —Macro-F1: 0.640985
• —Micro-F1: 0.703438
• —Exact-set accuracy: 0.163701
• —Hamming loss: 0.245552

Files

• —adapter_model.safetensors — QLoRA adapter weights
• —adapter_config.json — PEFT adapter configuration
• —tokenizer/ — tokenizer and Gemma chat template saved with the run
• —resolved_config.yaml — full experiment configuration
• —thresholds.json — development-selected strategy thresholds
• —dev_summary.json — development-set results
• —test_summary.json — test-set results
• —dev_per_label.csv — development per-label results
• —test_per_label.csv — test per-label results
• —training_history.csv — training history
• —runtime.json — recorded runtime information
• —hf_reload_full_test_comparison.csv — Hugging Face reload validation
• —inference.py — benchmark-compatible model loading/scoring utilities

Important usage note

This repository contains a PEFT adapter rather than a merged copy of the Gemma 3 12B base model. Access to and use of the base model remain subject to the terms applicable to google/gemma-3-12b-it.

The classifier requires the candidate-label prompting formulation used by the E2R benchmark. Loading the adapter alone as a generic text classification pipeline does not reproduce the task.

Quick start

The classifier takes a standard sentence and its Easy-to-Read rewrite and predicts all six transformation strategies independently.

Installation

pip install torch transformers peft bitsandbytes accelerate huggingface_hub

Gemma 3 access must be available for the Hugging Face account used to run the model.

Classify one sentence pair

from inference import E2RStrategyClassifier

classifier = E2RStrategyClassifier.from_pretrained(
    "hannah-khallaf/e2r-strategy-gemma-3-12b-pairwise-qlora"
)

result = classifier.predict(
    standard=(
        "Before moving to Spain, Kate had worked "
        "as a barrister in the UK."
    ),
    easy_to_read=(
        "Kate was a lawyer in the United Kingdom. "
        "Then she moved to Spain."
    ),
)

print(result["labels"])

for label, probability in result["probabilities"].items():
    print(
        f"{label:20s} "
        f"{probability:.4f} "
        f"(threshold={result['thresholds'][label]:.2f})"
    )

The result has the form:

{
    "labels": [
        # strategies predicted for this pair
    ],
    "probabilities": {
        "Synonymy": ...,
        "Modulation": ...,
        "Compression": ...,
        "Explanation": ...,
        "Syntactic Change": ...,
        "Omission": ...
    },
    "thresholds": {
        "Synonymy": 0.54,
        "Modulation": 0.87,
        "Compression": 0.11,
        "Explanation": 0.16,
        "Syntactic Change": 0.32,
        "Omission": 0.11
    }
}

The values above are deliberately left unspecified here. Use the model output itself for the probabilities; the classifier applies the development-selected threshold for each strategy automatically.

Batch prediction

examples = [
    {
        "standard":
            "Before moving to Spain, Kate had worked "
            "as a barrister in the UK.",
        "easy_to_read":
            "Kate was a lawyer in the United Kingdom. "
            "Then she moved to Spain.",
    },
    {
        "standard":
            "Well, uh, I am hungry.",
        "easy_to_read":
            "I am hungry.",
    },
]

results = classifier.predict_batch(examples)

for result in results:
    print(result["labels"])

Output interpretation

A probability should not be interpreted using a universal threshold of 0.5. Each strategy uses the threshold selected on the development set.

For example, a Compression probability of 0.20 is classified as present because the Compression threshold is 0.11, while a Modulation probability of 0.80 is classified as absent because its threshold is 0.87.

The model performs six candidate-label evaluations for every sentence pair, so several strategies may be returned simultaneously.