ThisenEkanayake/UltraChat-Sinhala

Dataset Card for UltraChat-Sinhala

Dataset Description

UltraChat-Sinhala is a Sinhala (සිංහල) machine translation of [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k), built to supervised-fine-tune Sinhala large language models. It preserves the original dataset's structure, splits, and prompt_ids, so it is a drop-in Sinhala counterpart to the English source.

The English dialogues were translated with NLLB-200-3.3B (eng_Latn → sin_Sinh) and then put through a Sinhala-specific cleaning pipeline (conjunct/ZWJ repair, masking-leak repair, filtering). The dataset contains 515,211 dialogues / ≈528M Sinhala tokens (SinLLaMA tokenizer).

Dataset Creation

1. 1.Translation. Each dialogue turn was translated en→si with NLLB-200-3.3B (greedy decoding). To respect the model's 512-token limit, turns were sentence-segmented with the line layout preserved on reassembly. Spans that must not be translated — fenced/inline code, URLs, e-mail addresses, HTML tags, LaTeX/maths and markdown links — were masked before translation and restored afterwards, so they pass through verbatim.

1. 1.Sinhala conjunct (ZWJ) repair. NLLB's SentencePiece normaliser strips the Zero-Width Joiner (U+200D) from Sinhala conjunct clusters, emitting a space instead (e.g. ප් ර for ප්‍ර). A lexicon-gated restorer — built from a Sinhala corpus plus the tokenizer's vocabulary — re-inserts the joiner only where attested. After repair, ~99.6–99.9% of dialogues carry conjunct joiners.

1. 1.Masking-leak repair. The placeholder used to mask the protected spans was itself corrupted by SentencePiece (its rare brackets were stripped), displacing each masked span to the end of its message with a stray digit left behind. These were re-inserted in their correct positions by re-aligning each message to its English source (≈83% reconstructed exactly in place; the rest re-appended without any content loss).

1. 1.Filtering. Dialogues containing empty/whitespace-only turns were dropped (a mid-conversation turn cannot be removed without breaking the user/assistant alternation); prompt_ids were de-duplicated; and one prompt_id shared between the SFT and GEN sets was removed for global id-uniqueness.

1. 1.Train/test split. Membership is taken verbatim from the original ultrachat_200k split by prompt_id, so the Sinhala split is identical to the English source (≈10% test). It is leak-checked: no prompt_id and no identical dialogue appears in both train and test, in either SFT or GEN.

Dataset Structure

Like the source, the dataset has four splits, suitable for:

• —Supervised fine-tuning (sft).
• —Generation ranking (gen) via techniques like rejection sampling or PPO.

Tokens are raw message-content tokens (the SinLLaMA tokenizer, vocab 139,336, has no chat template; a real SFT run adds a small per-turn special-token overhead).

The dataset is stored in parquet (zstd), schema-identical to the source:

{
    "prompt": "ආහාර පිසීමේ ව්‍යාපාරයක් සඳහා වට්ටෝරු පොතක් නිර්මාණය කරන්න. ...",
    "prompt_id": "7d86ffeefdea030c92138e0b964c304508bfebed4b23261c8a741630823e6f96",
    "messages": [
        {
            "role": "user",
            "content": "ආහාර පිසීමේ ව්‍යාපාරයක් සඳහා වට්ටෝරු පොතක් නිර්මාණය කරන්න. ..."
        },
        {
            "role": "assistant",
            "content": "නම: රසවත් ආහාර පිසීමේ රහස්: රසවත් හා ලස්සන ආහාර පිසීමේ වට්ටෝරු පොත ..."
        },
        {
            "role": "user",
            "content": "ඔයා මේ වෙනකම් හදපු වට්ටෝරු පොත නම් නියමයි. ඔයාට පුලුවන්ද තව විස්තර එකතු කරන්න ..."
        },
        {
            "role": "assistant",
            "content": "..."
        }
    ]
}

Quality and Limitations

Full-scan checks on the released data: 100% valid JSON, correct user/assistant alternation, no empty turns, no duplicate prompt_ids. ~0.017% of turns remain in Latin script (overwhelmingly fenced code, which is preserved by design), and ~0.5% of longer turns show NLLB repetition artifacts.

This is machine translation without human post-editing. Expect:

• —translationese and occasional mistranslation, especially on idioms, named entities, and technical content;
• —code, identifiers, URLs and maths intentionally left in their original form;
• —a minority of turns with NLLB repetition loops (~0.5%);
• —residual orthographic edge cases not covered by the conjunct lexicon.

It is intended for instruction-tuning and research, not as a gold-standard reference translation.

Licensing

Released under the MIT License, following the source dataset HuggingFaceH4/ultrachat_200k. The Sinhala text was produced by machine translation with NLLB-200.