mephistosir329/Aksharantar

Dataset Card for Aksharantar

Table of Contents

• —Table of Contents
• —Dataset Description
• —Dataset Summary
• —Supported Tasks and Leaderboards
• —Languages
• —Dataset Structure
• —Data Instances
• —Data Fields
• —Data Splits
• —Dataset Creation
• —Curation Rationale
• —Source Data
• —Annotations
• —Personal and Sensitive Information
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Additional Information
• —Dataset Curators
• —Licensing Information
• —Citation Information
• —Contributions

Dataset Description

• —Homepage: https://indicnlp.ai4bharat.org/indic-xlit/
• —Repository: https://github.com/AI4Bharat/IndicXlit/
• —Paper: Aksharantar: Towards building open transliteration tools for the next billion users
• —Leaderboard:
• —Point of Contact:

Dataset Summary

Aksharantar is the largest publicly available transliteration dataset for 20 Indic languages. The corpus has 26M Indic language-English transliteration pairs.

Supported Tasks and Leaderboards

[More Information Needed]

Languages

Dataset Structure

Data Instances

A random sample from Hindi (hin) Train dataset.

{
'unique_identifier': 'hin1241393', 
'native word': 'स्वाभिमानिक', 
'english word': 'swabhimanik', 
'source': 'IndicCorp', 
'score': -0.1028788579
}

Data Fields

• —unique_identifier (string): 3-letter language code followed by a unique number in each set (Train, Test, Val).
• —native word (string): A word in Indic language.
• —english word (string): Transliteration of native word in English (Romanised word).
• —source (string): Source of the data.
• —score (num): Character level log probability of indic word given roman word by IndicXlit (model). Pairs with average threshold of the 0.35 are considered.

For created data sources, depending on the destination/sampling method of a pair in a language, it will be one of:

• —Dakshina Dataset
• —IndicCorp
• —Samanantar
• —Wikidata
• —Existing sources
• —Named Entities Indian (AK-NEI)
• —Named Entities Foreign (AK-NEF)
• —Data from Uniform Sampling method. (Ak-Uni)
• —Data from Most Frequent words sampling method. (Ak-Freq)

Data Splits

Dataset Creation

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Who are the source language producers?

[More Information Needed]

Annotations

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Annotation process

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Who are the annotators?

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

<!-- <a rel="license" float="left" href="http://creativecommons.org/publicdomain/zero/1.0/"> <img src="https://licensebuttons.net/p/zero/1.0/88x31.png" style="border-style: none;" alt="CC0" width="100" /> <img src="https://mirrors.creativecommons.org/presskit/buttons/88x31/png/by.png" style="border-style: none;" alt="CC-BY" width="100" href="http://creativecommons.org/publicdomain/zero/1.0/"/> </a> <br/> -->

This data is released under the following licensing scheme:

• —Manually collected data: Released under CC-BY license.
• —Mined dataset (from Samanantar and IndicCorp): Released under CC0 license.
• —Existing sources: Released under CC0 license.

CC-BY License

<a rel="license" float="left" href="https://creativecommons.org/about/cclicenses/"> <img src="https://mirrors.creativecommons.org/presskit/buttons/88x31/png/by.png" style="border-style: none;" alt="CC-BY" width="100"/> </a>

<br> <br> <!-- and the Aksharantar benchmark and all manually transliterated data under the Creative Commons CC-BY license (“no rights reserved”). -->

CC0 License Statement

<a rel="license" float="left" href="https://creativecommons.org/about/cclicenses/"> <img src="https://licensebuttons.net/p/zero/1.0/88x31.png" style="border-style: none;" alt="CC0" width="100"/> </a>

<br> <br>

• —We do not own any of the text from which this data has been extracted.
• —We license the actual packaging of the mined data under the Creative Commons CC0 license (“no rights reserved”).
• —To the extent possible under law, <a rel="dct:publisher" href="https://indicnlp.ai4bharat.org/aksharantar/"> <span property="dct:title">AI4Bharat</span></a> has waived all copyright and related or neighboring rights to <span property="dct:title">Aksharantar</span> manually collected data and existing sources.
• —This work is published from: India.

Citation Information

@misc{madhani2022aksharantar,
      title={Aksharantar: Towards Building Open Transliteration Tools for the Next Billion Users}, 
      author={Yash Madhani and Sushane Parthan and Priyanka Bedekar and Ruchi Khapra and Anoop Kunchukuttan and Pratyush Kumar and Mitesh Shantadevi Khapra},
      year={2022},
      eprint={},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contributions