Datasetpublic

MLDataScientist/oasst2_uzbek

Open Assistant Conversations Dataset Release 2 (OASST2) in Uzbek language This dataset is an Uzbek translated version of OASST2 dataset. Llama3 chat template + thread formatted dataset based on this translation is also available for model fine-tuning here. The Uzbek translation was completed in 45 hours using a single T4 GPU and nllb-200-3.3B model. Based on nllb metrics, you might want to only filter out records that were not originally in English or Russian since… See the full description on the dataset page: https://huggingface.co/datasets/MLDataScientist/oasst2_uzbek.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

2likes18downloads

Dataset Card

Open Assistant Conversations Dataset Release 2 (OASST2) in Uzbek language

This dataset is an Uzbek translated version of OASST2 dataset.

Llama3 chat template + thread formatted dataset based on this translation is also available for model fine-tuning here.

The Uzbek translation was completed in 45 hours using a single T4 GPU and nllb-200-3.3B model.

Based on nllb metrics, you might want to only filter out records that were not originally in English or Russian since English-Uzbek and Russian-Uzbek have acceptable metrics and translation quality is noticeable better for those pairs based on my short reviews.

I am sharing the entire Uzbek translated dataset for future research.

The following repo and command was used to do the Uzbek translation.

Repo: https://github.com/UnderstandLingBV/LLaMa2lang

Command used:

!python3 translate.py nllb --model_size 3.3B uzn_Latn output_uzbek --quant8 --base_dataset OpenAssistant/oasst2 --max_length 512 --checkpoint_n 400 --batch_size 40```


I will fine-tune LLAMA3 8B Uzbek chat model and release in HF soon.