Modelpublic

omargrist/gemma-4-e2b-besstie-sarcasm-en-au

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes9downloads

Model Card

Gemma 4 E2B BESSTIE Sarcasm - en-AU

This repository contains a LoRA adapter for google/gemma-4-E2B, fine-tuned for binary sarcasm detection on the en-AU portion of surrey-nlp/BESSTIE-CW-26.

The adapter is intentionally variety-specific. During training and validation it only saw en-AU examples, rather than examples from the other BESSTIE varieties. This makes it useful for cross-variety transfer experiments: for example, evaluating an en-AU adapter on en-IN and en-UK test examples.

Model summary

—Base model: google/gemma-4-E2B
—Adapter method: QLoRA PEFT adapter
—Task: binary sarcasm detection
—Output labels: Yes = sarcastic, No = not sarcastic
—Training variety: en-AU
—Training seed used for this adapter: 100
—Dataset: surrey-nlp/BESSTIE-CW-26

This is not a merged standalone model. It contains the PEFT adapter weights and configuration, and should be loaded on top of the Gemma 4 E2B base model.

Dataset and training data

The adapter was fine-tuned on surrey-nlp/BESSTIE-CW-26, a benchmark for sentiment and sarcasm classification across English varieties. The dataset includes text, variety, source, sentiment, and sarcasm fields.

For this adapter, only the en-AU subset was used for training and validation.

Prompt format

The adapter was trained as a next-token prompted classifier. Use the same prompt format at inference time:

text

Detect sarcasm in the following English user-generated text.
Return Yes if sarcastic and No if not sarcastic.

Text: <text>

Label:

The model was trained to generate one of two single-token labels:

— No
— Yes

Training procedure

The adapter was trained using the following setup:

Setting	Value
Base model	`google/gemma-4-E2B`
Dataset	`surrey-nlp/BESSTIE-CW-26`
Training variety	`en-AU`
Seed used for release	`100`
Epochs	`4`
Learning rate	`2e-4`
Per-device batch size	`4`
Gradient accumulation	`4`
Warmup ratio	`0.05`
Weight decay	`0.01`
Scheduler	cosine
Max sequence length	`512`
LoRA rank	`8`
LoRA alpha	`16`
LoRA dropout	`0.10`
Quantisation	4-bit NF4 with double quantisation
Compute dtype	`bfloat16`
Optimised metric during training	validation loss
Checkpoint selection	best checkpoint by validation loss

Loading the adapter

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model = "google/gemma-4-E2B"
adapter_id = "omargrist/gemma-4-e2b-besstie-sarcasm-en-au"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, use_fast=True)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=quant_config,
    dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

Inference example

python

import torch

NEGATIVE_LABEL = " No"
POSITIVE_LABEL = " Yes"

def make_prompt(text):
    return (
        "Detect sarcasm in the following English user-generated text.\n"
        "Return Yes if sarcastic and No if not sarcastic.\n\n"
        f"Text: {text}\n\n"
        "Label:"
    )

def label_token_ids(tokenizer):
    no_ids = tokenizer(NEGATIVE_LABEL, add_special_tokens=False)["input_ids"]
    yes_ids = tokenizer(POSITIVE_LABEL, add_special_tokens=False)["input_ids"]

    if len(no_ids) != 1 or len(yes_ids) != 1:
        raise ValueError("Expected single-token labels for No/Yes.")

    return no_ids[0], yes_ids[0]

@torch.inference_mode()
def predict_sarcasm_prob(model, tokenizer, text):
    id_no, id_yes = label_token_ids(tokenizer)
    prompt = make_prompt(text)

    batch = tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=512,
        add_special_tokens=False,
    ).to(model.device)

    outputs = model(**batch, logits_to_keep=1)
    logits = outputs.logits[:, -1, :]
    label_logits = logits[:, [id_no, id_yes]]

    return torch.softmax(label_logits.float(), dim=-1)[0, 1].item()

text = "Oh brilliant, another meeting that could have been an email."
prob_sarcastic = predict_sarcasm_prob(model, tokenizer, text)

print({"prob_sarcastic": prob_sarcastic})

Results

The training script evaluates all 3 adapters on:

1.the full BESSTIE test set;
2.each individual test variety: en-AU, en-IN, and en-UK.

The decision threshold is tuned on the target validation split for the adapter variety.

Adapter	Test subset	Macro-F1
`en-AU`	en-AU	0.767
`en-AU`	en-IN	0.495
`en-AU`	en-UK	0.602

Citation

If you use this adapter, please cite the BESSTIE dataset paper referenced by the dataset card:

bibtex

@misc{besstie2024,
  title={BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English},
  year={2024},
  eprint={2412.04726},
  archivePrefix={arXiv}
}