innadark/topxgen-gemma-3-27b-and-nllb-3.3b

TopXGen: Topic-Diverse Parallel Data for Low-Resource MT

Dataset Summary

This dataset is a synthetic parallel dataset for 10 low-resource languages, created by applying the TopXGen pipeline with recent multilingual LLMs. It is designed for machine translation (MT) fine-tuning and few-shot experiments (as a selection pool). The pipeline works as follows:

1. 1.Topic-diverse paragraph generation in the target low-resource language using an LLM (generator), with diversity controlled via topic selection and temperature settings.
2. 2.Sentence splitting followed by translation/back-translation with a MT model (back-translator).
3. 3.Redundancy removal similar to the self-instruct approach.

Models trained on this TopXGen dataset achieve translation performance close to that of the generator and back-translator. For more details, see our paper.

Supported Languages

• —Basque (eus)
• —Hausa (hau)
• —Igbo (ibo)
• —Kinyarwanda (kin)
• —Nepali (nep)
• —Somali (som)
• —Sundanese (sun)
• —Swahili (swh)
• —Urdu (urd)
• —Xhosa (xho)

Data Generation

• —Generator: gemma-3-27b-it
• —Back-translator: nllb-200-3.3B

Example Usage

from datasets import load_dataset

dataset = load_dataset("almanach/topxgen-gemma-3-27b-and-nllb-3.3b", split="Basque")
print(dataset)

Output

Dataset({
    features: ['source', 'target', 'source_language', 'target_language'],
    num_rows: 120031
})

Licensing

This dataset is derived from outputs of Google’s Gemma-3 and Meta’s NLLB. Users must comply with the licenses and usage guidelines of both models.