Hassaanatif992/code-switching-codesaviours-si26-MuhammadHassaan

Roman Urdu-English Language Identification Model

Project Information

Project: Code Saviours SI-26 — Week 7 Intern: Muhammad Hassaan Student ID: SI26-ML-MH-004 Department: AI/ML

Model Description

This model is a fine-tuned XLM-RoBERTa model for word-level language identification in Roman Urdu-English code-switched text.

The model predicts one of three labels for each word:

• —URD — Roman Urdu
• —ENG — English
• —MIX — Mixed Roman Urdu-English token

Base Model

The model was fine-tuned from:

xlm-roberta-base

Dataset

The original dataset contained 150 Roman Urdu-English code-switched sentences.

During Week 7, additional MIX examples were added because the original dataset contained very few MIX-labelled tokens.

The augmented dataset contained 180 sentences.

The dataset was divided into:

• —144 training sentences
• —36 testing sentences

Training

XLM-RoBERTa was fine-tuned for token classification using GPU training.

Training configuration included:

• —Epochs: 5
• —Learning rate: 2e-5
• —Training batch size: 16
• —Evaluation strategy: Every epoch
• —Labels: URD, ENG, MIX

Because the MIX class was underrepresented, sentences containing MIX-labelled tokens were oversampled during the final training experiment.

Evaluation

The model was evaluated using per-class F1 scores for:

• —URD
• —ENG
• —MIX

The final balanced model performed better on the MIX class than the original training experiment.

Important: The MIX test set contained only a small number of MIX-labelled examples, so the MIX F1 score should be interpreted carefully.

Final F1 Scores

URD F1 : 0.9865 ENG F1 : 0.9853 MIX F1 : 1.0000 Macro F1 : 0.9906 Accuracy : 0.9863

Example

Input:

Aaj mera meeting schedule bohot busy hai

Expected language identification:

• —Aaj → URD
• —mera → URD
• —meeting → ENG
• —schedule → ENG
• —bohot → URD
• —busy → ENG
• —hai → URD

Mixed-token example:

Mujhe fileupload karni hai

• —Mujhe → URD
• —fileupload → MIX
• —karni → URD
• —hai → URD

Limitations

• —The dataset is relatively small.
• —The MIX class has fewer examples than URD and ENG.
• —Roman Urdu spelling is highly variable.
• —Performance may decrease on unseen spellings or sentence styles.
• —The model should be considered a research and internship project rather than a production-ready language identification system.

Future Improvements

Future work could include:

• —Collecting a larger Roman Urdu-English dataset.
• —Adding more naturally occurring MIX examples.
• —Improving class balance.
• —Testing on a larger unseen dataset.
• —Performing additional hyperparameter tuning.

Author

Muhammad Hassaan Student ID: SI26-ML-MH-004 Code Saviours SI-26