abdul-razaq-ork/romanized-pashto-fasttext

license: apache-2.0 language:

• —ps
• —en
• —ur tags:
• —fasttext
• —text-classification
• —language-identification
• —pashto
• —romanized-pashto
• —low-resource-language
• —code-mixed
• —nlp
• —language-detection ---

Romanized Pashto FastText Language Detector

Romanized Pashto FastText Language Detector

A lightweight FastText-based language identification model for detecting Romanized Pashto text.

The model performs binary classification between:

• —pashto
• —not_pashto

It is designed specifically for Romanized Pashto, where Pashto is written using Latin characters rather than the traditional Pashto script.

The model is intended for research and practical NLP applications involving low-resource languages, Romanized text, and code-mixed social-media language.

Model Details

Model Description

This model uses FastText supervised text classification to identify whether a given input is likely to be Romanized Pashto.

The model was developed to address the difficulty of identifying Romanized Pashto in digital text, where users may write Pashto using Latin characters and frequently mix Pashto with English or Urdu.

The negative class contains non-Pashto examples, including English and Roman Urdu.

• —Developed by: Abdul Razaq
• —Model type: FastText supervised text classifier
• —Task: Binary language identification
• —Languages: Romanized Pashto, English, Roman Urdu
• —Labels: pashto, not_pashto
• —License: Apache-2.0
• —Finetuned from: Not applicable

Model Sources

• —Repository: This Hugging Face model repository
• —Paper: Not currently available
• —Demo: Not currently available

Uses

Direct Use

The model can be used for:

• —Romanized Pashto language identification
• —Filtering Romanized Pashto from multilingual datasets
• —Corpus construction
• —Data preprocessing
• —Social-media text analysis
• —Code-mixed language processing
• —Low-resource NLP research
• —Identifying candidate Romanized Pashto text for manual annotation

Downstream Use

The model can be integrated into larger NLP pipelines for:

• —Romanized Pashto sentiment analysis
• —Text classification
• —Dataset cleaning
• —Language filtering
• —Corpus construction
• —Social-media NLP
• —Human annotation workflows
• —Low-resource language research

For example, the model can be used as an initial filtering step before applying a more computationally expensive NLP model.

Out-of-Scope Use

The model should not be used as the sole basis for high-stakes decisions.

It should not be used for:

• —Determining a person's ethnicity or nationality
• —Determining a person's identity
• —Legal or immigration decisions
• —Employment decisions
• —Educational decisions
• —Automatic deletion or moderation without human review
• —Treating model predictions as linguistic ground truth

The model is specifically designed for Romanized Pashto identification and should not be assumed to provide reliable identification of standard Pashto written in the Pashto/Arabic-derived script.

Bias, Risks, and Limitations

Romanized Pashto does not have a universally standardized spelling system.

The same Pashto word may be written in several different ways using Latin characters. This creates challenges for automatic language identification.

Performance may be affected by:

• —Spelling variation
• —Transliteration variation
• —Informal writing
• —Very short text
• —Code-mixing
• —Roman Urdu
• —English loanwords
• —Unseen vocabulary
• —Dialectal variation
• —Social-media-specific language

Because the not_pashto class includes English and Roman Urdu, the model may perform differently on languages that were not sufficiently represented in the training and evaluation data.

The reported results should therefore not be interpreted as universal performance across all Romanized Pashto.

Recommendations

Users should consider model confidence when interpreting predictions.

For research applications, it is recommended to:

1. 1.Select an appropriate confidence threshold.
2. 2.Manually inspect uncertain predictions.
3. 3.Evaluate the model on data from the target domain.
4. 4.Report class distributions.
5. 5.Avoid treating automatically generated labels as ground truth without validation.

How to Get Started

Install FastText:

pip install fasttext

Load the model:

import fasttext

model = fasttext.load_model("model.bin")

Run a prediction:

text = "za kha yam"

labels, probabilities = model.predict(text)

print(labels)
print(probabilities)

Example output:

(('__label__pashto',), array([0.98]))

The model uses the following labels:

__label__pashto
__label__not_pashto

The returned probability represents the model's confidence in the prediction.

Training Details

Training Data

The model was trained as a binary language-identification classifier using:

Positive class

Romanized Pashto

Pashto text represented using Latin characters.

Negative class

Non-Pashto text, including:

• —English
• —Roman Urdu

The dataset was developed for research into Romanized Pashto and low-resource language processing.

The data includes informal language characteristics relevant to digital communication.

Training Procedure

The model uses FastText supervised classification.

FastText uses word and subword information to represent text. This is particularly useful for Romanized language varieties where spelling and word forms can vary substantially.

Preprocessing

Text preprocessing was performed before model training.

The preprocessing pipeline was designed to prepare the text for FastText classification while retaining useful lexical and character-level information.

Training Hyperparameters

• —Training framework: FastText
• —Training method: Supervised classification
• —Task: Binary classification
• —Classes: pashto, not_pashto
• —Model architecture: FastText

Exact FastText training hyperparameters are not included in this version of the Model Card.

Evaluation

Testing Data

The model was evaluated on a held-out dataset containing:

The evaluation data contains Romanized Pashto and non-Pashto examples, with English and Roman Urdu represented in the negative class.

Evaluation Metrics

The following metrics were used:

• —Accuracy
• —Precision
• —Recall
• —F1-score
• —Macro F1
• —Weighted F1
• —Balanced Accuracy

F1-score provides a balance between precision and recall.

Macro F1 gives equal weight to each class, while weighted F1 accounts for the number of examples in each class.

Balanced accuracy is useful when comparing performance across classes.

Results

At the default classification threshold, the model achieved:

Class-Level Results

The model achieved very high precision for the pashto class, while its recall for Pashto was comparatively lower at the default threshold.

This indicates that predictions classified as Romanized Pashto were highly precise on the reported evaluation set, although some Romanized Pashto examples were missed.

Threshold Analysis

A threshold analysis was performed to examine how changing the decision threshold affects model performance.

The best observed threshold for Pashto F1-score was approximately:

0.01

At this threshold:

These results demonstrate that classification performance can change substantially depending on the decision threshold.

Users should select a threshold according to the requirements of their particular application.

For example:

• —A higher threshold may prioritize precision.
• —A lower threshold may improve recall.

Model Examination

The model was examined using class-level precision, recall, F1-score, and threshold analysis.

The results show a clear precision-recall trade-off.

At the default threshold, the model provides very high precision for the Romanized Pashto class.

Threshold analysis shows that adjusting the decision threshold can improve recall and balanced performance.

Future evaluation may include:

• —Cross-validation
• —Confidence calibration
• —Statistical significance testing
• —Error analysis
• —Evaluation on additional Romanized Pashto varieties
• —Evaluation on larger multilingual datasets

Environmental Impact

FastText is a lightweight NLP architecture and can perform inference efficiently on CPU-based systems.

Carbon emissions can be estimated using the Machine Learning Impact calculator presented by Lacoste et al. (2019).

• —Hardware Type: Not reported
• —Hours used: Not reported
• —Cloud Provider: Not reported
• —Compute Region: Not reported
• —Carbon Emitted: Not calculated

Technical Specifications

Model Architecture and Objective

The model uses FastText supervised text classification.

FastText uses word and subword information to construct representations of text.

This makes FastText useful for:

• —Noisy text
• —Spelling variation
• —Short text
• —Low-resource languages
• —Romanized language varieties

The classification objective is:

Input Text
     |
     v
FastText
     |
     v
Binary Classifier
     |
     +--------> pashto
     |
     +--------> not_pashto

Compute Infrastructure

The model is lightweight and can be used on standard CPU-based systems.

Hardware

Training hardware is not reported.

Software

The model can be used with the FastText implementation and Python.

Citation

If you use this model in academic research, please cite this model repository and the associated research work when available.

BibTeX

@misc{abdul_razaq_romanized_pashto_fasttext,
  author = {Abdul Razaq},
  title = {Romanized Pashto FastText Language Detector},
  year = {2026},
  publisher = {Hugging Face},
  note = {FastText-based Romanized Pashto language identification model}
}

APA

Abdul Razaq. (2026). Romanized Pashto FastText Language Detector. Hugging Face.

Glossary

Romanized Pashto: Pashto represented using Latin characters instead of the traditional Pashto script.

Language Identification: The task of determining which language a text belongs to.

Code-mixing: The use of multiple languages within the same communication or text.

FastText: A lightweight NLP framework that uses word and subword information for text representation and classification.

Macro F1: The average F1-score across classes, giving each class equal importance.

Weighted F1: The average F1-score across classes weighted according to the number of examples in each class.

Balanced Accuracy: The average recall across the classes.

More Information

This model is part of ongoing work in:

• —Romanized Pashto NLP
• —Low-resource language processing
• —Code-mixed language processing
• —Language identification
• —Computational linguistics
• —NLP dataset development

Future versions may include:

• —Larger datasets
• —More diverse Romanized Pashto examples
• —Additional transliteration variants
• —More negative-language classes
• —Improved threshold calibration
• —Cross-validation
• —Statistical evaluation
• —Error analysis
• —Integration with downstream Romanized Pashto NLP systems

Model Card Authors

Abdul Razaq

Model Card Contact

For questions, suggestions, or collaboration related to this model, please use the Discussions section of this Hugging Face repository.