Amna-Shahzad/code-switching-codesaviours-si26-Amna

---
language:
- en
- ur
tags:
- token-classification
- code-switching
- roman-urdu
- urdu
- english
- multilingual
- xlm-roberta
- text-classification
- code-mixing
license: apache-2.0
base_model: FacebookAI/xlm-roberta-base
pipeline_tag: token-classification
---

# Urdu-English Code-Switching Detection

A fine-tuned **XLM-RoBERTa** model for identifying whether individual words in code-switched text are **Urdu, English, or Mixed**.

This model was developed as part of the **CodeSaviours SI-26** project by **Amna Shahzad** at **Code Saviours (SMC-PRIVATE) Limited**.

---

## Overview

Code-switching is the practice of using words or expressions from multiple languages within the same sentence or conversation.

In social media and informal communication, Urdu and English are frequently mixed together. This is especially common in:

- YouTube comments
- WhatsApp messages
- Twitter/X posts
- Instagram content

For example:

> `Aaj weather bohat amazing hai`

A human reader can easily understand the sentence, but automatically determining the language of each word can be challenging.

This model addresses this problem by classifying individual words into three categories:

| Label | Meaning |
|------|---------|
| `URD` | Urdu |
| `ENG` | English |
| `MIX` | Mixed Urdu-English / code-switched word |

---

# Project Objective

The primary objective of this project is to develop a language identification model capable of detecting the language category of individual words in **Urdu-English code-switched text**.

The model can be used as a component in larger NLP systems for:

- Code-switching analysis
- Urdu-English language identification
- Social media text analysis
- Multilingual NLP
- Roman Urdu processing
- Language-aware text preprocessing
- Code-mixed sentiment analysis
- Urdu-English text normalization

---

# Model

The model is based on:

**XLM-RoBERTa Base**

XLM-RoBERTa is a multilingual Transformer model designed to process text across multiple languages.

For this project, the pretrained model was fine-tuned using a **token-classification architecture**.

### Architecture

The model performs classification at the token/word level rather than assigning a single label to the complete sentence.

---

# Label Definitions

The model uses three labels:

### `URD`

Represents words identified as Urdu.

Example:

### `ENG`

Represents English words.

Example:

### `MIX`

Represents words that contain or combine Urdu and English characteristics.

Example:

may be classified as:

> **Note:** The exact behavior of `MIX` depends on the annotation guidelines used during dataset creation.

---

# Dataset

The model was trained on a manually collected dataset of **175 sentences**.

| Split     | Number of Sentences |
| --------- | ------------------: |
| Training  |                 140 |
| Testing   |                  35 |
| **Total** |             **175** |

The dataset consists of Urdu-English code-switched sentences collected from multiple real-world communication platforms.

### Data Sources

The collected examples were sourced from:

* YouTube comments
* WhatsApp messages
* Twitter/X posts
* Instagram content

The purpose of using multiple sources was to capture different styles of informal Urdu-English communication.

---

# Example

An example of code-switched text:

Possible token-level classification:

Another example:

Possible classification:

---

# Training

The pretrained `xlm-roberta-base` model was fine-tuned for token classification.

### Training Configuration

| Parameter             | Value                |
| --------------------- | -------------------- |
| Base Model            | `xlm-roberta-base`   |
| Task                  | Token Classification |
| Number of Labels      | 3                    |
| Training Samples      | 140                  |
| Testing Samples       | 35                   |
| Epochs                | 5                    |
| Training Batch Size   | 16                   |
| Evaluation Batch Size | 8                    |
| Learning Rate         | `5e-5`               |
| Optimizer             | AdamW                |
| LR Scheduler          | Linear               |
| Weight Decay          | 0.0                  |
| Warmup Steps          | 0                    |
| Gradient Accumulation | 1                    |
| Random Seed           | 42                   |
| Evaluation Strategy   | Epoch                |
| Save Strategy         | Epoch                |
| Best Model Selection  | Validation Loss      |

---

# 📈 Performance

The model achieved an overall token-level accuracy of:

## **99.70%**

This result was obtained on the held-out test set containing **35 sentences**.

| Metric           |     Result |
| ---------------- | ---------: |
| Overall Accuracy | **99.70%** |
| Training Samples |        140 |
| Testing Samples  |         35 |

### Interpretation

An overall accuracy of **99.70%** means that approximately 99.7% of the evaluated tokens were assigned the correct language label according to the test annotations.

> Because the dataset is relatively small, this result should be interpreted cautiously. Evaluation on a larger and more diverse dataset would provide a stronger estimate of real-world generalization.

---

# Intended Use

This model is intended for research and educational purposes involving:

* Urdu-English code-switching
* Roman Urdu
* Multilingual NLP
* Token-level language identification
* Social media text analysis
* Code-mixed NLP pipelines

It can also be used as a preprocessing component for downstream applications such as:

---

# How to Use

Install the required libraries:

Then load the model:

Run the model on a sentence:

Example output:

> The exact output may vary depending on tokenization and model predictions.

---

# Methodology

The overall development process consisted of the following stages:

### 1. Data Collection

Code-switched Urdu-English sentences were collected from real-world sources including social media and messaging platforms.

### 2. Data Preparation

The collected sentences were organized and annotated according to the three target categories:

### 3. Dataset Split

The dataset was divided into:

### 4. Tokenization

The text was tokenized using the XLM-RoBERTa tokenizer.

### 5. Label Alignment

Labels were aligned with the corresponding subword tokens generated by the tokenizer.

### 6. Fine-Tuning

`xlm-roberta-base` was fine-tuned using the prepared training dataset.

### 7. Evaluation

The trained model was evaluated on the held-out test set.

### 8. Final Performance

The model achieved:

**99.70% overall token-level accuracy.**

---

# Limitations

Although the model achieved high accuracy on the current test set, there are several limitations to consider.

### Small Dataset

The dataset contains only **175 sentences**, which is relatively small for training and evaluating a robust multilingual NLP system.

### Limited Test Set

Only **35 sentences** were used for testing. Therefore, the reported 99.70% accuracy may not fully represent performance on unseen real-world data.

### Domain Dependence

The data was collected primarily from informal communication platforms. Performance may differ on:

* Formal Urdu
* News articles
* Academic text
* Long documents
* Historical Urdu
* Highly specialized terminology

### Roman Urdu Variation

Roman Urdu does not have a single standardized spelling system. The same Urdu word may be written in several different ways.

For example:

Such variations can make language identification more difficult.

### Mixed Words

Words combining Urdu and English can be ambiguous and may require contextual understanding.

---

# Future Work

Several improvements can be explored in future versions:

### Larger Dataset

Increase the dataset from hundreds of sentences to thousands or more.

### More Data Sources

Include additional sources such as:

* Reddit
* Public forums
* News comments
* Online discussions

### Improved Annotation

Develop more detailed annotation guidelines for ambiguous and mixed words.

### Model Comparison

Compare XLM-RoBERTa with other multilingual and language-specific Transformer models.

### Additional Metrics

Evaluate the model using:

* Precision
* Recall
* F1-score
* Per-class performance
* Confusion matrix

### Data Augmentation

Generate additional realistic variations of Roman Urdu and code-switched sentences.

### Better Generalization

Evaluate the model on completely unseen domains and larger external datasets.

---

# Repository Information

**Model:**
`Amna-Shahzad/code-switching-codesaviours-si26-Amna`

**Base Model:**
`xlm-roberta-base`

**Task:**
Token Classification

**Labels:**

**Training Dataset:**
140 sentences

**Testing Dataset:**
35 sentences

**Overall Accuracy:**
99.70%

---

# Author

**Amna Shahzad**

### Project

**CodeSaviours SI-26**

### Organization

**Code Saviours (SMC-PRIVATE) Limited**

---

# License

This model is intended for research and educational use.

Please review the licensing terms of the underlying `xlm-roberta-base` model and any datasets used before using this model in a commercial application.

---

# Acknowledgements

This project builds upon the work of the XLM-RoBERTa authors and the open-source Hugging Face Transformers ecosystem.

Special thanks to the developers and researchers who make multilingual NLP models and tools publicly available.

---

# References

* XLM-RoBERTa: *Unsupervised Cross-lingual Representation Learning at Scale*
* Hugging Face Transformers documentation
* Hugging Face Datasets ecosystem

---