Zerothe00/code-switched-student-blindspot-eval

Code Switched Student Blind Spot Evaluation

Overview

This repository contains a small manual evaluation of Qwen/Qwen2.5-1.5B-Instruct on code-switched South Asian international student prompts. The goal is to test whether a small open-weight instruction model can understand Pakistani English mixed with Roman Urdu/Hindi in situations shaped by scholarship pressure, family expectations, limited resources, and international student life in Malaysia.

Blind Spot

Current AI models often respond well to standard English prompts, but they can struggle when the real meaning depends on local language, indirect phrasing, family obligations, scholarship anxiety, or low-resource student constraints. These situations are common for students from Pakistan and South Asia studying abroad, but they are rarely tested in standard model benchmarks.

The evaluation focuses on four categories:

1. 1.Scholarship search and pressure
2. 2.Family expectations and emotional pressure
3. 3.Using AI as an international student in Malaysia
4. 4.Low-resource research and student life

Model

Model tested: Qwen/Qwen2.5-1.5B-Instruct

I selected this model because it is an open-weight instruction model in the required 0.6B to 6B parameter range. It is small enough to run in Google Colab while still being strong enough to reveal meaningful behavior.

Dataset

The dataset contains 32 prompts, with 8 prompts per category. Each row includes:

• —prompt id
• —category
• —prompt
• —expected good answer
• —what to check
• —model response
• —score
• —notes

The prompts use English, Pakistani English, and Roman Urdu/Hindi expressions such as "ghar walon ka pressure," "scene ye hai," "paisay," "samajh nahi aa raha," and "fully funded scholarship."

Scoring

Each model response was manually scored:

• —2: good response. The model understood the mixed language and student context and gave practical advice.
• —1: partial response. The model understood the broad topic but missed important cultural, financial, academic, or code-switched context.
• —0: failed response. The model misunderstood the prompt, hallucinated, ignored the real constraint, or gave irrelevant or risky advice.

Results

Overall mean score: 0.719 / 2

Score distribution:

• —2: 4 responses
• —1: 15 responses
• —0: 13 responses

Category summary:

Key Findings

The model often understood the broad topic of a prompt, but it struggled with the actual lived context behind the words.

Examples of failures:

• —It misunderstood "fee waiver" and hallucinated that "waafa" meant "no application fee."
• —It misread "SOP" as "Software Process Improvement" instead of Statement of Purpose.
• —It treated "kabhi kabhi university me" as a university name instead of "sometimes at university."
• —It refused a safe question about whether AI should be used to understand visa and part-time work rules.
• —It gave generic GPU/cloud advice instead of practical low-resource AI research paths.
• —It often missed family pressure, scholarship anxiety, and the need to verify official sources.

These failures suggest that the model can understand some surface-level English, but it is less reliable when meaning depends on code-switching, social pressure, or local student realities.

Proposed Path Forward

A better model should be trained and evaluated on more realistic international student communication. Useful improvements could include:

1. 1.Curating code-switched South Asian student prompts with Roman Urdu/Hindi and Pakistani English.
2. 2.Including scenarios about scholarships, visa rules, family expectations, low-resource research, and university administration.
3. 3.Adding human-written reference answers from students or mentors familiar with these contexts.
4. 4.Fine-tuning or instruction-tuning models to ask clarifying questions when local terms or institutional rules are ambiguous.
5. 5.Building a benchmark that rewards practical, culturally aware, and safety-conscious advice rather than generic fluency.

Files

• —code_switched_student_blindspot_dataset.csv: original prompt dataset
• —qwen2_5_1_5b_code_switched_student_outputs_reviewed.csv: model outputs with manual scores and notes
• —qwen2_5_1_5b_code_switched_student_summary.csv: category-level score summary
• —qwen_code_switched_student_eval_colab.ipynb: notebook used to run the model in Colab