BeitTigreAI/tigre-data-kenLM

Tigre 5-gram Language Model (KenLM)

Overview

This repository provides a 5-gram Language Model (LM) for the Tigre language, trained using the KenLM toolkit. This model is a foundational resource for various downstream NLP and speech applications, including:

• —Rescoring hypotheses in Automatic Speech Recognition (ASR).
• —Improving text generation and fluency in Machine Translation (MT).
• —Performing basic text filtering and quality control.

The model is provided in the standard ARPA format, and can be converted to a compact binary format for efficient use in production environments.

Model Statistics

Training Data Source

This model was trained on a cleaned Tigre monolingual text dataset, combining Tigre-language news text with additional narrative text, to give the model broader coverage than news alone.

More detailed information about the training data can be found in the dataset's documentation: Tigre Monolingual Text Dataset README

Files and Structure

tigre-data-kenLM/ ├── README.md └── tigre-data-kenLM.arpa

How to Use the Model

You can load and query the model using the Python bindings for KenLM (kenlm).

Installation

pip install kenlm huggingface_hub

Example Usage (Perplexity and Score)

import kenlm
from huggingface_hub import hf_hub_download

# 1. Download the ARPA model file from the Hugging Face Hub
arpa_path = hf_hub_download(
    repo_id="BeitTigreAI/tigre-data-kenLM",
    filename="tigre-data-kenLM.arpa",
    repo_type="model"
)

# 2. Load the KenLM model
lm = kenlm.Model(arpa_path)

# Example sentence to score
test_sentence = "ህተ ትብል ሑዬቱ ወአነ እብል ሕቼተ"

# A. Log10 probability of the entire sentence
log_prob = lm.score(test_sentence)
print(f"Sentence: '{test_sentence}'")
print(f"Log10 Probability: {log_prob:.4f}")

# B. Perplexity of the entire sentence
perplexity = lm.perplexity(test_sentence)
print(f"Perplexity: {perplexity:.2f}")

Licensing and Citation

The Tigre 5-gram Language Model is licensed under CC-BY-SA-4.0.

Recommended Citation Format

Repository Name: Tigre 5-gram Language Model (KenLM) Organization: BeitTigreAI URL: https://huggingface.co/BeitTigreAI/tigre-data-kenLM