hassan-wajid/Spatial-Blind-Spots-in-Vision-Language-Models

license: mit

model_evaluated:

• —name: Qwen3-VL-2B-Instruct url: https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct

evaluation_notebook:

• —https://www.kaggle.com/code/wajidhassanmoosa/blind-spot-qwen3-2b

evaluationsetup: | The model evaluated in this study is Qwen3-VL-2B-Instruct. Evaluation was conducted using the Hugging Face Transformers library with automatic device mapping (devicemap="auto") and "bfloat16" dtype selection.

For each example:

1. 1.The image was provided as part of a structured chat message.
2. 2.A reasoning-oriented spatial question was appended.
3. 3.The processor applied the official chat template.
4. 4.The model generated an open-ended answer.
5. 5.The output was compared against the expected spatial ground truth.

Generation was performed with maxnewtokens=32 and no additional prompt engineering beyond concise response constraints.

Full evaluation code is available at: https://www.kaggle.com/code/wajidhassanmoosa/blind-spot-qwen3-2b

blindspotdescription: | While Qwen3-VL-2B-Instruct performs strongly on descriptive multimodal tasks, it exhibits consistent spatial reasoning blind spots under controlled evaluation.

Observed systematic errors include:

1. 1.Dynamic State Tracking Failures: The model fails to correctly update object relationships after hypothetical manipulations (e.g., swapping, lifting, sliding).

1. 1.Reflection Geometry Confusion: The model misinterprets mirrored or perspective-based spatial relationships, particularly in side-view mirrors and reflective surfaces.

1. 1.Mental Rotation and Horizontal Flip Errors: When asked to imagine flipped or rotated configurations, the model often preserves original spatial relationships incorrectly.

1. 1.Containment and Physical Support Failures: The model confuses inside/outside relationships and support structures, suggesting limited internal physical simulation.

1. 1.Small Object Counting Inaccuracy: The model frequently miscounts small identical objects under minimal clutter.

These patterns indicate that large-scale web pretraining prioritizes caption alignment over structured spatial reasoning.

finetuningrecommendation: | To mitigate these blind spots, fine-tuning should focus on structured spatial reasoning supervision rather than generic caption datasets.

The required dataset should emphasize:

1. 1.Counterfactual Spatial Manipulation:
2. 2.Object swaps
3. 3.Hypothetical repositioning
4. 4.Horizontal/vertical flips
5. 5.Step-wise state transitions

1. 1.Physically Grounded Containment:
2. 2.Inside/outside reasoning
3. 3.Support/contact relationships
4. 4.Object dependency chains

1. 1.Reflection-Aware Reasoning:
2. 2.Mirror perspective inversion
3. 3.Transparent surfaces
4. 4.Occluded reflections

1. 1.Controlled Counting Tasks:
2. 2.Identical object sets
3. 3.Progressive object addition/removal
4. 4.Varying background complexity

Crucially, prompts must require reasoning, not simple recognition or captioning.

datasetconstructionstrategy: | A scalable training corpus can be assembled through a hybrid strategy:

(A) Synthetic Scene Generation (Primary Recommendation)

• —Use Blender or programmatic 2D engines.
• —Control object positions precisely.
• —Automatically compute ground-truth spatial relationships.
• —Generate transformation variants (flip, swap, move).

(B) Procedural Augmentation

• —Apply geometric transformations to base scenes.
• —Automatically recompute spatial labels.
• —Ensure category-balanced sampling.

(C) Curated Real-World Collection

• —Mirror-based scenes
• —Transparent containers
• —Support/stacking configurations
• —Annotated with structured reasoning prompts

Synthetic generation is preferred because:

• —It ensures exact spatial supervision.
• —It reduces annotation noise.
• —It enables scalable controlled counterfactual variations.

estimateddatasetsize: | To meaningfully improve spatial reasoning performance:

• —5,000 examples: Minimal measurable improvement in isolated categories.

• —20,000–50,000 examples: Moderate robustness and improved generalization across transformations.

• —100,000+ examples: Strong generalization across diverse spatial compositions, especially if category-balanced.

Each failure category should contain at least 5,000–10,000 diverse examples to avoid shortcut learning.