hamnaheh/code-switching-codesaviours-si26-humna

Code Switching NLP Dataset — Code Saviours SI-26, Humna Imran

Word-level labeled dataset of naturally code-switched Roman Urdu + English sentences, as commonly written by Pakistani social media users.

What it is

220 sentences (2,490 word-level rows). Each word is labeled URD (Roman Urdu), ENG (English), or MIX (hybrid token, e.g. hyphenated compounds).

How it was built

1. 1.Source sentences come from Smat26/Roman-Urdu-Dataset (GitHub), a public, UCI-referenced compilation of ~20,000 Roman Urdu sentences gathered from Twitter, Facebook comments, and e-commerce reviews (credit: Zareen Sharf).
2. 2.Sentences were filtered to keep only genuine code-switching: at least 2 English words and 3 Urdu words, with Urdu still the majority language.
3. 3.Word-level labels were assigned automatically via dictionary lookup against the top 10,000 common English words (google-10000-english), with a curated override list for Roman Urdu words that collide with English spellings (e.g. "mil", "beta", "par"). This mirrors the method used in prior Roman-Urdu code-switching research.
4. 4.A random sample was manually spot-checked and corrected.

Label meanings

• —URD: Roman Urdu word
• —ENG: English word
• —MIX: hybrid/hyphenated token combining both

Limitations

Automatic dictionary-based labeling can misclassify short words that exist in both languages by coincidence (e.g. "is", "to"). A curated override list reduces but does not eliminate this. Treat labels as high-quality silver annotations, not gold-standard.

Credit

Source sentences: Smat26/Roman-Urdu-Dataset (GPL-3.0), compiled by Zareen Sharf, referenced at the UCI Machine Learning Repository.