Alaa-Abdelkader/fatima-fellowship-qwen35-eval

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3.5-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)


## Blind Spot Analysis (Observations from Qwen3.5-4B)

Based on my testing, this model exhibits three distinct types of blind spots:

1. **Tokenization and Counting:** The model incorrectly counted 2 'a's in Alaa Salah, and 3 "n"s in Banana. This happens because models see the word as a single token rather than individual letters.
2. **Infinite Thinking Loops:** On complex constraint tasks (like the 5-word cat sentence), the model's "thinking process" logic gets stuck and fails to provide a final answer string.
3. **Temporal Hallucination:** The model incorrectly placed the invention of the internet before the moon landing, showing a failure in chronological fact retrieval.
4. **Base Model Repetition:** On simple tasks like the Fibonacci sequence, the model provides the answer but then fails to "stop," repeating the user's prompt (Eos-token failure).

## Fine-Tuning Strategy
To fix these, I recommend a Supervised Fine-Tuning (SFT) dataset of 25,000 examples featuring:
- **Chain-of-Thought (CoT)** reasoning for logic puzzles.
- **Negative constraint** training (e.g., "write without using X").
- **Strict output formatting** to prevent the model from repeating the prompt or getting stuck in internal logs.