CoolFace
Modelpublic

ogx786/urdu-roman-transliterator-tiny-aya-V3

sourceHugging Facecc-by-nc-4.0updated 3mo agoView on Hugging Face
0likes
Model Card

Urdu → Roman Urdu Transliterator (Tiny Aya, QLoRA fine-tune)

This model transliterates Urdu script text into Roman Urdu (Urdu written in Latin script), fine-tuned from `CohereLabs/tiny-aya-global` using QLoRA.

Unlike translation, this is a transliteration task: the output preserves the original words and meaning, just rendered in Latin script instead of Urdu script.

Example:

Urdu InputRoman Urdu Output
کیا حال ہے؟kya haal hai?

Model Details

  • —Base model: `CohereLabs/tiny-aya-global` (3.35B parameters, Cohere2 architecture)
  • —Fine-tuning method: QLoRA (4-bit quantized base, LoRA adapters merged into full weights for this release)
  • —Task: Urdu script → Roman Urdu transliteration
  • —License: CC-BY-NC-4.0, inherited from the base model, plus Cohere Labs' Acceptable Use Policy. Non-commercial use only.

Training Data

Trained on ~80,000 Urdu ↔ Roman Urdu sentence pairs, combining two sources:

Split as 70,000 train / 5,000 validation / 5,000 test, deduplicated and shuffled before splitting.

Training Procedure

  • —Method: QLoRA (4-bit NF4 base model, LoRA adapters on attention projections)
  • —Epochs: 1
  • —Effective batch size: 16 (batch size 4 × gradient accumulation 4)
  • —Learning rate: 2e-4
  • —Warmup steps: 200
  • —Optimizer: pagedadamw8bit
  • —Precision: fp16
  • —Max sequence length: 224 tokens
  • —Loss masking: completion-only loss — the instruction and input portion of each example is masked out (label = -100), so the model is only trained to predict the Roman Urdu output, not to reproduce the prompt.

<!-- Optional: add your eval numbers here once available, e.g.

Evaluation

MetricScore
BLEU...
BERTScore F1...

Evaluated on the held-out 5,000-example test split. -->

Prompt Format

This model was trained on a strict instruction format. Inference must match this format exactly, or output quality will degrade:

### Instruction:
Transliterate the following Urdu text into Roman Urdu. 
Output ONLY the Roman Urdu. No translation. No explanation.
### Input:
{urdu_text}
### Response:

The model was trained to generate only the Roman Urdu text after ### Response:\n, followed by an end-of-sequence token. It was not trained to add explanations, translations, or any other content.

Inference

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "ogx786/urdu-roman-transliterator-tiny-aya-V3"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()

PROMPT_TEMPLATE = """### Instruction:
Transliterate the following Urdu text into Roman Urdu. 
Output ONLY the Roman Urdu. No translation. No explanation.
### Input:
{urdu_text}
### Response:
"""

def transliterate(urdu_text: str, max_new_tokens: int = 128) -> str:
    prompt = PROMPT_TEMPLATE.format(urdu_text=urdu_text)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,          # deterministic output for transliteration
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.eos_token_id,
        )

    # Decode only the newly generated tokens, not the prompt
    new_tokens = output_ids[0][inputs["input_ids"].shape[1]:]
    result = tokenizer.decode(new_tokens, skip_special_tokens=True)

    # Safety net: if the model ever continues past the response
    # (e.g. starts a new "### Instruction:" block), cut it off there.
    result = result.split("### Instruction:")[0].split("### Input:")[0].strip()
    return result


if __name__ == "__main__":
    examples = [
        "کیا حال ہے؟",
        "میں ٹھیک ہوں، شکریہ۔",
    ]
    for urdu in examples:
        print(f"Urdu:  {urdu}")
        print(f"Roman: {transliterate(urdu)}")
        print()

Batch inference

For transliterating many sentences, batch the prompts together rather than calling generate in a loop:

python
def transliterate_batch(urdu_texts: list[str], max_new_tokens: int = 128) -> list[str]:
    prompts = [PROMPT_TEMPLATE.format(urdu_text=u) for u in urdu_texts]
    tokenizer.padding_side = "left"  # required for correct batched generation
    inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)

    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.eos_token_id,
        )

    results = []
    for i in range(len(urdu_texts)):
        new_tokens = output_ids[i][inputs["input_ids"].shape[1]:]
        text = tokenizer.decode(new_tokens, skip_special_tokens=True)
        text = text.split("### Instruction:")[0].split("### Input:")[0].strip()
        results.append(text)
    return results

Intended Use

  • —Transliterating Urdu script text (names, addresses, sentences) into Roman Urdu for downstream systems that expect Latin-script input (e.g. voice assistants, search, SMS-based systems).
  • —Research on low-resource script conversion for South Asian languages.

Limitations

  • —Trained on 1 epoch over ~70K examples; performance on domains very different from the training data (e.g. poetry, heavy code-switching, informal social media text) is not guaranteed.
  • —do_sample=False (greedy decoding) is recommended for this task since transliteration should be deterministic; sampling may introduce inconsistency.
  • —Non-commercial use only, per the base model's CC-BY-NC-4.0 license and Cohere's Acceptable Use Policy.

Citation

If you use this model, please cite the base model:

@misc{tinyaya,
  title = {Cohere Labs Tiny Aya},
  author = {Cohere Labs},
  year = {2026},
  url = {https://huggingface.co/CohereLabs/tiny-aya-global}
}