ahmad0999/qwen-blindspot-symbolic-reasoning

Qwen3.5-0.8B Blind Spot Dataset

Multi-Step Symbolic Reasoning Under Linguistic Camouflage

Overview

This dataset documents a targeted blind spot of [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), a compact 0.8-billion parameter instruction-tuned language model released by the Qwen team (Alibaba Cloud) in March 2026.

Final score: 5 / 10 correct.

The central finding is not that the model cannot reason — it visibly tries on every single probe, and gets the arithmetic right more often than not. The finding is more specific: the model's final answer is sometimes generated independently of its own reasoning trace. In several failures the model writes a correct chain of logic and then directly contradicts it in the conclusion. This "last-mile disconnect" is the blind spot this dataset is designed to characterize.

Model Tested

How We Loaded the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "Qwen/Qwen3.5-0.8B"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",   # bfloat16 on CUDA, float32 on CPU
    device_map="auto",
)
model.eval()

def ask(prompt: str, max_new_tokens: int = 256) -> str:
    """Single-turn non-thinking chat inference."""
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,   # non-thinking mode — default for 0.8B
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.7,
            top_p=0.8,
            top_k=20,
            do_sample=False,   # greedy — fully deterministic outputs
        )
    new_ids = output_ids[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(new_ids, skip_special_tokens=True).strip()

Dependencies:

pip install transformers torch accelerate datasets pandas

Dataset Schema

Results

5 / 10 correct.

The Story the Data Tells

Reading the full model outputs — not just the final answers — reveals four distinct, mechanistically different failure modes. They are worth examining individually because they have different causes and require different fixes.

Failure Mode 1 — Conclusion Flip (probes 02, 09)

This is the most striking failure and the theoretical core of this dataset. The model produces a fully correct reasoning trace and then contradicts it in the final sentence.

Probe 09 is the clearest example. The model writes:

"Since Zara is a blork, and all blorks are snorps, Zara is a snorp. Since Zara is a snorp, and no snorps are frimbles, Zara is not a frimble."

And then immediately concludes:

"Yes, Zara is a frimble."

Every step of the logic is correct. The conclusion is a direct contradiction of the step immediately preceding it. The answer appears to have been generated from the surface framing of the question — a positive question ("Is Zara a frimble?") generating a positive answer — rather than from the logical content of the preceding tokens.

Probe 02 shows the same failure at the chain-assembly level. The model correctly writes S is lighter than P in step 3, but when building the combined inequality it writes P < S < Q < R, reversing the S–P relationship it had just established, and concludes P is lightest. The model knows the chain-assembly template; it made a transcription error that went uncorrected because there is no verification pass.

Interpretation: In non-thinking mode, the conclusion token is generated autoregressively like any other token. The model cannot re-read its scratchpad and check for consistency. The phrase "Therefore, the answer is X" is a high-frequency conclusion template in instruction-tuning data, and the model sometimes fills it from distributional priors anchored in the question's surface features (positive phrasing, most recently mentioned entity) rather than from the semantic content of the preceding reasoning.

Failure Mode 2 — Remainder Ignored (probe 04)

The model correctly computes 75 ÷ 12 = 6 cycles with 3 hours remaining — the operative modular step — and then discards the remainder entirely. It enumerates all six cycles as 11:00, states that "75 hours is exactly 6 cycles," and concludes the clock shows 11:00.

This is distinct from the conclusion-flip failure. The model did not misread its trace — it actively discarded a computed intermediate value when completing a template. The enumeration of "Cycle 1: 11:00, Cycle 2: 11:00 …" suggests the model shifted into a cycle-listing template and resolved the final slot by pattern continuation rather than by returning to the arithmetic. The remainder, once generated, was not carried forward.

Failure Mode 3 — Missing Division (probe 05)

The model correctly identifies that each of 6 people shakes hands with 5 others. It explicitly states the correct justification — "every handshake involves two people" — and then multiplies 6 × 5 = 30 without dividing by 2. It verbally acknowledged the symmetry correction and did not apply it numerically.

This is a well-documented failure mode for small models on combinatorics: retrieving the verbal rule ("handshakes involve two people") does not guarantee applying the corresponding arithmetic operation (÷2). The rule is generated as a statement — a token sequence — not as a trigger for a computation. The model proceeds to the multiplication template before the division is applied.

Failure Mode 4 — Off-by-One in Counting (probe 03)

The modular arithmetic is correct: 100 mod 7 = 2. The failure is in interpreting "2 days forward from Wednesday." The model counts:

"1. Wednesday, 2. Thursday — therefore Thursday."

It counts Wednesday itself as day 1, arriving one day early. This is a classic fence-post error that arises when "the current position" and "the first step of counting" are conflated. The model knows to add 2 days; it counts inclusively from the starting position rather than exclusively from it.

What the Correct Answers Tell Us

The five passes are informative too. Probes 06 and 07 involve multi-step arithmetic that the model executes cleanly — including correctly flooring 10 ÷ 3 = 3 in probe 06. Probe 08, the hardest arithmetic probe (compound percentage change), is solved completely correctly: four symbolic steps, correct sign, correct conclusion, correct percentage stated. Probe 10 (chained conditionals with familiar vocabulary) is handled correctly. Probe 01 (the forward transitive chain) is correct.

This pattern shows the model is not uniformly weak at symbolic reasoning. It fails specifically when: (a) a sign or value must be faithfully transcribed across multiple representation steps without drift (probes 02, 04), (b) a symmetry correction must be numerically applied after a verbal rule is retrieved (probe 05), (c) a conclusion template is filled from surface cues instead of from the trace content (probe 09), or (d) an inclusive/exclusive counting boundary must be correctly set (probe 03). In the absence of these specific traps, the model reasons well.

Discussion: What Fine-Tuning Dataset Would Fix This?

The specific fix each failure mode requires

Conclusion flip (probes 02, 09): Training examples where the conclusion sentence is required to explicitly restate the key intermediate result before asserting the answer. For example: "Since the chain gives S < P < Q < R and S is the leftmost, the answer is S." This forces the model to bridge its trace and its final token rather than filling a conclusion template independently.

Remainder ignored (probe 04): Modular problems where the remainder is non-zero and the final step explicitly names it: "The remainder is 3. We add 3 hours to 11:00 to get 14:00, which is 2:00 on a 12-hour clock." The remainder must be referenced by name in the answer, not left as a discarded intermediate.

Missing division (probe 05): Combinatorics problems where the verbal rule and the arithmetic are paired immediately: "Since each handshake is shared between two people, the 30 counted interactions double-count every handshake, so the true count is 30 ÷ 2 = 15." The division must appear in the same sentence as the justification, not deferred to a later step.

Off-by-one (probe 03): Date and sequence problems with explicit exclusive-start framing: "We do not count Wednesday itself. Day 1 is Thursday, day 2 is Friday. The answer is Friday."

What kind of dataset, and how big?

The ideal fine-tuning dataset is a chain-of-thought SFT corpus with grounded conclusions — every conclusion sentence must name the intermediate result it derives from, not merely state an answer. The failures here are not knowledge failures; they are failures of trace-to-conclusion coherence. The dataset must specifically supervise that interface.

Relevant existing sources to mine and extend:

• —GSM8K — grade-school math with step-by-step traces; strong base for probes 06–08 style
• —LogiQA — logical reasoning; covers probe 09/10 style
• —bAbI tasks — synthetic chained reasoning; directly relevant to probe 01/02
• —Synthetically generated (problem, trace, answer) triples from a larger model (e.g., Qwen3.5-9B), filtered for correctness and audited for conclusion-trace consistency

Estimated sizes:

The key design principle is quality over scale: one example where the conclusion is explicitly grounded in a named intermediate step is worth more than ten examples that only supervise the final answer token. For a 0.8B model, careful SFT on ~5,000 well-designed examples is likely to move all four failure modes more than 50,000 examples of answer-only supervision.

Citation

@dataset{qwen_blindspot_2026,
  title     = {Qwen3.5-0.8B Blind Spot: Multi-Step Symbolic Reasoning},
  year      = {2026},
  note      = {Evaluation dataset for Qwen/Qwen3.5-0.8B},
  url       = {https://huggingface.co/datasets/YOUR_USERNAME/qwen-blindspot-symbolic-reasoning}
}

License

Probe inputs and annotations: CC BY 4.0. Model outputs reproduced for research and evaluation purposes only.