NLPC-UOM/sentence_alignment_dataset-Sinhala-Tamil-English

Dataset summary

This is a gold-standard benchmark dataset for sentence alignment, between Sinhala-English-Tamil languages. Data had been crawled from the following news websites. The aligned documents annotated in the dataset NLPC-UOM/documentalignmentdataset-Sinhala-Tamil-English had been considered to annotate the aligned sentences.

The aligned sentences have been manually annotated.

Dataset

The folder structure for each news source is as follows.

si-en
  |--army
      |--Sinhala
      |--English
      |--army.si-en
  |--hiru <br/>
      |--Sinhala 
      |--English 
      |--hiru.si-en
  |--itn 
      |--Sinhala 
      |--English 
      |--itn.si-en
  |--newsfirst
      |--Sinhala 
      |--English 
      |--newsfirst.si-en 
ta-en
si-ta

Sinhala/English/Tamil - contain the aligned documents in the two languages with respect to the news source. (army/hiru/itn/newsfirst) Aligned documents contain the same ID.<br/> army.si-en - golden aligned sentence alignment. Each sentence is referenced according to the languageprefixfileidsentenceId. <br/>

Citation Information

@article{fernando2022exploiting,<br/> title={Exploiting bilingual lexicons to improve multilingual embedding-based document and sentence alignment for low-resource languages},<br/> author={Fernando, Aloka and Ranathunga, Surangika and Sachintha, Dilan and Piyarathna, Lakmali and Rajitha, Charith},<br/> journal={Knowledge and Information Systems},<br/> pages={1--42},<br/> year={2022},<br/> publisher={Springer}<br/> }