agentlans/multilingual-e5-small-aligned-quality

multilingual-e5-small-aligned-quality

This model is a fine-tuned version of agentlans/multilingual-e5-small-aligned designed for assessing text quality across multiple languages.

Key Features

• —Multilingual support
• —Quality assessment for text
• —Based on E5 small model architecture

Intended Uses & Limitations

This model is intended for:

• —Assessing the quality of multilingual text
• —Filtering multilingual content
• —Comparative analysis of corpus text quality across different languages

Limitations:

• —Performance may vary for languages not well-represented in the training data
• —Should not be used as the sole criterion for quality assessment

Usage Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "agentlans/multilingual-e5-small-aligned-quality"

# Initialize tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

def quality(text):
    """Assess the quality of the input text."""
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True).to(device)
    with torch.no_grad():
        logits = model(**inputs).logits.squeeze().cpu()
    return logits.tolist()

# Example usage
score = quality("Your text here.")
print(f"Quality score: {score}")

Performance Results

The model was evaluated on a diverse set of multilingual text samples:

• —10 English text samples of varying quality were translated into Arabic, Chinese, French, Russian, and Spanish.
• —The model demonstrated consistent quality assessment across different languages for the same text.

<details> <summary>Click here for the 10 original texts and their translations.</summary>

</details>

<img src="Quality.svg" alt="Scatterplot of predicted quality scores grouped by text sample and language" width="100%"/>

Training Data

The model was trained on the Multilingual Parallel Sentences dataset, which includes:

• —Parallel sentences in English and various other languages
• —Semantic similarity scores calculated using LaBSE
• —Additional quality metrics
• —Sources: JW300, Europarl, TED Talks, OPUS-100, Tatoeba, Global Voices, and News Commentary

Training Procedure

Hyperparameters

• —Learning rate: 5e-05
• —Train batch size: 128
• —Eval batch size: 8
• —Seed: 42
• —Optimizer: AdamW (betas=(0.9,0.999), epsilon=1e-08)
• —Learning rate scheduler: Linear
• —Number of epochs: 3.0

Training Results

Framework Versions

• —Transformers: 4.46.3
• —PyTorch: 2.5.1+cu124
• —Datasets: 3.1.0
• —Tokenizers: 0.20.3