Tajwer/roman_urdu_negation_blindspot

Roman Urdu Negation Blind Spot

Dataset Summary

This dataset tests whether Qwen3-4B-Instruct-2507 uses negation correctly when judging Roman Urdu sentences, rather than just being able to translate it. It pairs 12 sentences across two polarities (affirmative and negated), tested on two tasks: English translation and a constrained Yes/No sentiment/factual judgment, with a matched English control set (48 rows total). Built as the technical challenge submission for the Fatima Fellowship.

Model Tested

• —Model: Qwen/Qwen3-4B-Instruct-2507
• —Parameters: 4B (dense, decoder-only, instruction-tuned)
• —License: Apache 2.0
• —Revision (pinned): cdbee75f17c01a7cc42f958dc650907174af0554
• —Model page: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

How the Model Was Loaded

!pip install -q -U transformers accelerate huggingface_hub

from huggingface_hub import model_info
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "Qwen/Qwen3-4B-Instruct-2507"
REVISION = model_info(MODEL_ID).sha   # pin to exact revision

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=REVISION)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, revision=REVISION, torch_dtype=torch.float16, device_map="auto"
)
model.eval()

Error Types

Why This Happens

A language model doesn't have a separate "judgment" step. It does one thing: predict the next token. Every answer it gives is a next-token prediction.

For Task B, the prompt ends with: "Question: Is the sentiment of this sentence positive? Answer with exactly one word: Yes or No." The model reads the whole prompt, and at the final position it produces a probability distribution over its entire vocabulary: every token it could emit next. The scoring code narrows that to two candidates, Yes and No. Whichever has the higher probability is the model's "answer." That's it: the "judgment" is just given everything in the prompt, which token is more likely to come next, Yes or No.

The model does not fail because it can't see negation. Task A shows it can: when the sentence says "nhi ha," the translation says "not." The negation word registers. The failure is downstream, in how the model uses that negation when asked to judge sentiment.

When the model reads "Wo ek buri sohbat mein hai" (he is in a bad company), it answers No (negative). That's correct, and correct because of the sentiment word "buri," with no negation involved. But when it reads "Wo ek buri sohbat mein nhi ha" (he is not in a bad company), the correct answer is Yes (positive), because "nhi" cancels "buri." The model still answers No. It is still following "buri," not processing "nhi." The negation is in the input but does not shift the answer.

The same failure appears in the opposite direction. "Film achi thi" (the film was good) is judged positive. That's correct. "Film achi nii thi" (the film was not good) should flip to negative, but the model still says positive. It latches onto "achi" and ignores "nii," exactly as it latched onto "buri" and ignored "nhi."

So the failure is not a bias toward negative answers, and not a failure to understand negation words. It is that the model answers from the sentiment word ("buri," "ganda," "achi") and lets the negation pass through without changing the judgment. The model is confidently wrong in these cases: the softmax probability for the chosen answer stays high, typically 0.90 and above, averaging 0.97 across the error set, even when the answer is incorrect. There is no internal signal that the negation was missed.

This is why the training fix has to target the judgment, not the vocabulary. The model already knows what "nhi" means. The data has to show it that the same sentence under two polarities gets two different labels (affirmative negative, negated positive) until the negation word carries weight in the final decision.

Fine-Tuning Recommendations

If the model couldn't see negation at all, the fix would be to show it more negation words and teach it what "nhi" means: translation-level training. But the model already knows what "nhi" means: it translates it correctly. The problem is downstream. When the model decides whether a sentence is positive or negative, it's not carrying the negation forward. It's latching onto the sentiment word ("buri," "ganda") and stopping there.

So the fix has to target the judgment, not the vocabulary: show the model the same sentence twice, same words, one with negation and one without, and flip the label (affirmative negative, negated positive). Same sentence, one word added, label flipped. Do that enough times and the model is forced to use the negation instead of ignoring it.

Recommended data: Start with ~1,000-2,000 Roman Urdu minimal pairs (affirmative/negated, label flipped) covering the negation constructions found in this probe: explicit (nhi, nii, ni, mt) and implicit ("forgot to" style, which flips the answer with no negation word present), split across both factual and sentiment judgment, matching the p-series/s-series split used here. This is a behavior correction on an already-pretrained model, not new vocabulary (Task A shows the model already knows what these words mean), so a narrow, targeted set should outweigh scale.

After fine-tuning, re-run this same probe methodology on a held-out set of pairs. If the negation-invariant rate hasn't dropped meaningfully, that's the signal to scale the data up rather than guessing a higher number up front.