Emmaka/qwen35-08b-base-blind-spots

Qwen3.5-0.8B-Base: Multi-Dimensional Blind Spot Dataset

Abstract

We present a structured dataset of blind spots discovered in [Qwen/Qwen3.5-0.8B-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base), a 0.8B parameter base language model released in March 2026. Using an automated pipeline grounded in three lines of NLP research — the reversal curse (Berglund et al., ICLR 2024), confidence calibration (Xiong et al., ICLR 2024), and behavioral testing (Ribeiro et al., ACL 2020) — we probed 45 facts across 186 total prompts and identified 129 errors (69.4% error rate).

Key findings:

• —Reversal curse is relation-dependent: Capital-country facts show a +40pp forward advantage, while author-work and company-founder facts show an inverted pattern (reverse is easier), revealing that the directionality of training data varies by relation type.
• —Severe overconfidence: The model maintains ~95% confidence regardless of correctness (ECE = 0.644), with all outputs in the [0.8, 1.0) confidence bin.
• —Language-bound knowledge: Only 4/45 facts are answered consistently across languages. English accuracy (43.3%) drops sharply in French (17.8%), Spanish (17.9%), and Portuguese (7.1%).

Model Under Test

[Qwen/Qwen3.5-0.8B-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)

• —0.8B parameter decoder-only language model
• —Base model (pretrained only — no instruction tuning or RLHF)
• —Released March 2, 2026 by Alibaba Qwen team
• —Supports 201 languages and dialects
• —Gated Delta Networks + sparse Mixture-of-Experts architecture
• —Apache 2.0 license

How the Model Was Loaded

The model was loaded in Google Colab (T4 GPU, 16GB VRAM) using HuggingFace Transformers in float16 precision:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "Qwen/Qwen3.5-0.8B-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

Generation used temperature=0.1, top_p=0.95, max_new_tokens=100 with output_scores=True to extract token-level log-probabilities for confidence analysis.

The full pipeline code (including prompt generation, confidence extraction, automated judging, and analysis) is available in the code/ directory of this dataset repository.

Methodology

Three Probe Dimensions

Each underlying fact was tested with 4-6 related prompts across three dimensions:

1. Forward vs. Reverse (Reversal Curse)

Following Berglund et al. (ICLR 2024), we test each fact in both directions:

• —Forward: "The capital of Japan is" → expect "Tokyo"
• —Reverse: "Tokyo is the capital of" → expect "Japan"

The reversal curse predicts that autoregressive models succeed on whichever direction appeared more frequently in training data. Our results show this is relation-dependent: some relation types show the classic forward advantage, while others are inverted.

2. Cross-Lingual Consistency (INV Test)

Following the invariance test methodology from CheckList (Ribeiro et al., ACL 2020), we ask the same factual question in English, French, Spanish, Portuguese, and German. A correctly-learned fact should be language-invariant.

3. Confidence Calibration

Following Xiong et al. (ICLR 2024) and the FermiEval benchmark (Epstein et al., 2025), we extract token-level log-probabilities to compute a normalized confidence score per generation.

Automated Evaluation

All 186 outputs were evaluated by Claude (claude-sonnet-4-20250514) using the LLM-as-Judge methodology (Zheng et al., NeurIPS 2023), with structured JSON output classifying each response as CORRECT, INCORRECT, or PARTIALLY_CORRECT with reasoning.

Fact Battery

45 facts across 11 relation types: capital-country (10), author-work (5), discoverer-discovery (3), inventor-invention (2), element-symbol (4), company-founder (4), country-language (3), person-achievement (4), math-science (5), river-location (2), currency-country (3).

Key Findings

Reversal Curse (+2.2pp aggregate gap)

• —Forward accuracy: 20/45 (44.4%)
• —Reverse accuracy: 19/45 (42.2%)
• —Aggregate gap not statistically significant (p=0.45), BUT:

The reversal asymmetry is strongly relation-dependent:

The inverted cases suggest the model's training data contained more "Person X founded/wrote/discovered Y" than "Y was founded/written/ discovered by Person X" — the reverse of the typical capital-country ordering. This extends Berglund et al.'s finding by showing that the reversal curse direction is predictable from training data patterns.

Confidence Calibration (ECE = 0.644)

• —Mean confidence when correct: 0.9606
• —Mean confidence when incorrect: 0.9486
• —Mann-Whitney U test: p = 0.085 (not significant)

The model maintains approximately 95% confidence on all outputs, regardless of correctness. All 186 prompts fell in the [0.8, 1.0) confidence bin. This extreme overconfidence is consistent with Xiong et al. (ICLR 2024) and the "perception tunnel" theory of Epstein et al. (2025).

Cross-Lingual Consistency (mean = 0.269)

• —Fully consistent facts: 4/45
• —English: 43.3% | French: 17.8% | Spanish: 17.9% | German: 44.4% | Portuguese: 7.1%

16 out of 45 facts are correct in English but fail in at least one other language. This demonstrates that factual knowledge is language-bound rather than stored in abstract, language-invariant representations — despite 201-language training.

Dataset Schema

Proposed Fine-Tuning Strategy

Based on the three failure dimensions identified, we recommend the following targeted fine-tuning approach:

1. Bidirectional Fact Training (to fix the Reversal Curse)

What: For every factual statement "A is B", include the reverse "B is A" as a separate training example with multiple paraphrases.

Source: Extract subject-object pairs from Wikidata and generate both orderings. E.g., from (Tokyo, capital_of, Japan), generate "The capital of Japan is Tokyo" AND "Tokyo is the capital of Japan" with 5-10 paraphrases each.

Size estimate: 5,000-10,000 bidirectional fact pairs (~50K-100K examples with paraphrases). Berglund et al. showed 30 facts x 30 paraphrases (900 examples) demonstrated the curse, so ~10K pairs should meaningfully reduce asymmetry.

Assembly: Wikidata SPARQL queries → (subject, relation, object) triples → template-based generation with teacher-model paraphrasing.

2. Multilingual Alignment Data (to fix Cross-Lingual Gaps)

What: Parallel factual statements in all target languages.

Source: Wikidata entities have labels in 200+ languages. Generate parallel statements from these. Supplement with MGSM and XL-Sum.

Size estimate: 3,000-5,000 parallel fact sets (each in 4-6 languages), totaling ~15K-30K training examples.

3. Calibration-Aware Training (to fix Overconfidence)

What: Contrastive examples where similar prompts have different answers, plus examples requiring hedged responses.

Source: TriviaQA with difficulty annotations; synthetically generated hard negatives via teacher model.

Size estimate: 2,000-5,000 calibration-focused examples.

Total Estimated Dataset Size

10,000-30,000 high-quality examples using LoRA fine-tuning (rank 8-16) to avoid catastrophic forgetting. At 0.8B parameters, full fine-tuning is feasible but LoRA preserves existing capabilities. Training: 3-5 epochs, ~1-2 hours on a single A100 GPU.

Limitations

• —Fact battery covers mainly encyclopedic knowledge; reasoning, code, and creative tasks not tested
• —Cross-lingual testing limited to 5 of 201 claimed languages
• —Automated judging via Claude may contain errors; spot-checking advised
• —Confidence scores from low-temperature (0.1) decoding may differ at higher temperatures
• —Per-relation reversal patterns based on small samples (2-10 per type)

References

1. 1.Berglund, L., et al. "The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A'." ICLR 2024.
2. 2.Ribeiro, M.T., et al. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." ACL 2020 (Best Paper).
3. 3.Xiong, K., et al. "Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs." ICLR 2024.
4. 4.Epstein, E.L., et al. "LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval." 2025.
5. 5.Zheng, L., et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023.

Citation

@dataset{qwen35_blind_spots_2026,
  author = {Emmaka, Imraul},
  title = {Qwen3.5-0.8B-Base Blind Spots: Reversal Curse, Calibration & Cross-Lingual Consistency},
  year = {2026},
  publisher = {Hugging Face},
}