CoolFace
Datasetpublic

belumind/en-vi-ja-curated-500k-triplets

EN-VI-JA Curated 500K Triplets High-quality English-Vietnamese-Japanese translation triplets curated from OPUS parallel corpora. Dataset Statistics Split Count Train 446,252 Validation 24,792 Test 24,792 Total 495,836 Quality Metrics Average Quality Score: 0.794 Average LaBSE Score: 0.859 LaBSE Threshold: 0.6 LaBSE Score Distribution Range Count 0.9-1.0 177,022 0.8-0.9 217,463 0.7-0.8 76,727… See the full description on the dataset page: https://huggingface.co/datasets/belumind/en-vi-ja-curated-500k-triplets.

sourceHugging Facecc-by-4.0updated 8mo agoView on Hugging Face
1likes438downloads
Dataset Card

EN-VI-JA Curated 500K Triplets

High-quality English-Vietnamese-Japanese translation triplets curated from OPUS parallel corpora.

Dataset Statistics

SplitCount
Train446,252
Validation24,792
Test24,792
Total495,836

Quality Metrics

  • —Average Quality Score: 0.794
  • —Average LaBSE Score: 0.859
  • —LaBSE Threshold: 0.6

LaBSE Score Distribution

RangeCount
0.9-1.0177,022
0.8-0.9217,463
0.7-0.876,727
0.5-0.724,624

Source Distribution

SourceCountPercentage
CCAligned370,87174.8%
HPLT98,41219.8%
WikiMatrix13,8872.8%
KDE48,3641.7%
LaboroParaCorpus1,9340.4%
Tatoeba1,1780.2%
Ubuntu1,1590.2%
GNOME11<0.1%
AMIMeeting11<0.1%
KFTT9<0.1%

Quality Pipeline (19 Steps)

V2 Steps (10)

  1. 1.Language validation (FastText + script)
  2. 2.Deduplication
  3. 3.Cross-lingual coherence
  4. 4.Text normalization
  5. 5.Content quality filters
  6. 6.Japanese tokenization
  7. 7.Quality-based selection
  8. 8.Source quality weighting
  9. 9.Triplet length ratio
  10. 10.Encoding handling

V3 Steps (9)

  1. 1.LaBSE semantic alignment
  2. 2.Broken English detection
  3. 3.Web artifact removal
  4. 4.B2B spam filtering
  5. 5.NSFW content filtering
  6. 6.Fragment detection
  7. 7.MT artifact detection
  8. 8.Script ratio validation
  9. 9.Character ratio validation

Quality Thresholds

ParameterValue
FastText confidence0.65
Vietnamese diacritic ratio0.02
Japanese script ratio0.10
LaBSE threshold0.60
Min tokens3
Max tokens100
Max length ratio2.5

Usage

python
from datasets import load_dataset

dataset = load_dataset("sotalab/en-vi-ja-curated-500k-triplets")

# Access splits
train = dataset["train"]
val = dataset["validation"]
test = dataset["test"]

# Example
print(train[0])
# {'en': '...', 'vi': '...', 'ja': '...', 'source': '...', 'quality_score': 0.85, 'labse_score': 0.92}

License

This dataset is released under the CC-BY-4.0 license.

Citation

If you use this dataset, please cite:

bibtex
@dataset{sotalab2024envija500k,
  title={EN-VI-JA Curated 500K Triplets},
  author={SotaLab},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/sotalab/en-vi-ja-curated-500k-triplets}
}