Rychanfox/semantic-highlighting-qwen2.5-1.5b-lora

Semantic Highlighting SLM (Qwen2.5-1.5B QLoRA)

A QLoRA adapter fine-tuning Qwen2.5-1.5B-Instruct to tag rhetorical-role spans in technical abstracts by generating the source text back with inline [Label]...[/Label] tags -- Contribution, Method, Result, Evidence, Limitation, FutureWork, Safety. Built as the generative comparison point against a fine-tuned token classifier in the same project, to empirically test whether a small generative model's flexibility beats a purpose-built classifier for this task.

Trained with QLoRA: 4-bit quantized frozen base weights + LoRA adapters on the attention/MLP projections, ~1.18% of parameters trainable (18.4M / 1.56B).

The honest result (read before using this for anything real)

This model produces better labels than the classifier when it works, but it is only reliable 43% of the time and ~20x+ slower:

"Faithful" means the tag-stripped output exactly matches the source text, character for character. The other 57% of the time, this model duplicates or drops small pieces of the source text while tagging it -- specific, diagnosed failure modes (not random paraphrasing), documented in the findings log. Because a highlighter that silently alters the text it's supposed to be annotating is a correctness problem, not just a quality one, the [classifier](https://huggingface.co/Rychanfox/semantic-highlighting-modernbert-crf) is the model actually used in the project's interactive demo. This adapter is published for the comparison itself, not as a recommended production choice.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

tokenizer = AutoTokenizer.from_pretrained("Rychanfox/semantic-highlighting-qwen2.5-1.5b-lora")
quant_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct", quantization_config=quant_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "Rychanfox/semantic-highlighting-qwen2.5-1.5b-lora")

Prompt format, faithfulness checking, and tag parsing utilities are in the project repo's slm/ module (build_messages, parse_tags, predict).

Training data

Trained on the same Rychanfox/semantic-highlighting-abstracts dataset as the classifier, serialized into the [Label]...[/Label] inline-tag format as the target completion (system prompt describes the label schema; loss is masked to the completion only, not the prompt).

Known limitations

• —Reliability: see the faithfulness result above. Duplication and omission of short phrases are the two dominant failure modes.
• —Latency: not real-time; unsuitable for interactive use as currently implemented (greedy decoding, no speculative/constrained decoding).
• —Evaluated on a small hand-corrected gold set (21 abstracts) -- treat numbers as directional.