cointegrated/panlex-meanings

Dataset Card for panlex-meanings

This is a dataset of words in several thousand languages, extracted from https://panlex.org.

Dataset Details

Dataset Description

This dataset has been extracted from https://panlex.org (the 20240301 database dump) and rearranged on the per-language basis.

Each language subset consists of expressions (words and phrases). Each expression is associated with some meanings (if there is more than one meaning, they are in separate rows).

Thus, by joining per-language datasets by meaning ids, one can obtain a bilingual dictionary for the chosen language pair.

• —Curated by: David Dale (@cointegrated), based on a snapshot of the Panlex database (https://panlex.org/snapshot).
• —Language(s) (NLP): The Panlex database mentions 7558 languages, but only 6241 of them have at least one entry (where entry is a combination of expression and meaning), and only 1012 have at least 1000 entries. These 1012 languages are tagged in the current dataset.
• —License: CC0 1.0 Universal License, as explained in https://panlex.org/license.

Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

• —Original website: https://panlex.org/
• —Paper: Kamholz, David, Jonathan Pool, and Susan M. Colowick. 2014. PanLex: Building a Resource for Panlingual Lexical Translation. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014).

Uses

The intended use of the dataset is to extract bilingual dictionaries for the purposes of language learning by machines or humans.

The code below illustrates how the dataset could be used to extract a bilingual Avar-English dictionary.

from datasets import load_dataset
ds_ava = load_dataset('cointegrated/panlex-meanings', name='ava', split='train')
ds_eng = load_dataset('cointegrated/panlex-meanings', name='eng', split='train')
df_ava = ds_ava.to_pandas()
df_eng = ds_eng.to_pandas()

df_ava_eng = df_ava.merge(df_eng, on='meaning', suffixes=['_ava', '_eng']).drop_duplicates(subset=['txt_ava', 'txt_eng'])
print(df_ava_eng.shape)
# (10565, 11)

print(df_ava_eng.sample(5)[['txt_ava', 'txt_eng', 'langvar_uid_ava']])
#           txt_ava txt_eng langvar_uid_ava
# 7921        калим     rug         ava-002
# 3279       хІераб     old         ava-001
# 41     бакьулълъи  middle         ava-000
# 9542        шумаш    nose         ava-006
# 15030     гӏащтӏи     axe         ava-000

Apart from these direct translations, one could also try extracting multi-hop translations (e.g. enrich the direct Avar-English word pairs with the word pairs that share a common Russian translation). However, given that many words have multiple meaning, this approach usually generates some false translations, so it should be used with caution.

Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

The dataset is split by languages, denoted by their ISO 639 codes. Each language might contain multiple varieties; they are annotated within each per-language split.

To determine a code for your language, please consult the https://panlex.org webside. For additional information about a language, you may also want to consult https://glottolog.org/.

Each split contains the following fields:

• —id (int): id of the expression
• —langvar (int): id of the language variaety
• —txt (str): the full text of the expression
• —txt_degr (str): degraded (i.e. simplified to facilitate lookup) text
• —meaning (int): id of the meaning. This is the column to join for obtaining synonyms (within a language) or translations (across languages)
• —langvar_uid (str): more human-readable id of the language (e.g. eng-000 stands for generic English, eng-001 for simple English, eng-004 for American English). These ids could be looked up in the language dropdown at https://vocab.panlex.org/.

Dataset Creation

This dataset has been extracted from https://panlex.org (the 20240301 database dump) and automatically rearranged on the per-language basis.

The rearrangement consisted of the following steps:

1. 1.Grouping together the language varieties from the langvar table with the same lang_code.
2. 2.For each language, selecting the corresponding subset from the expr table.
3. 3.Joining the selected set with the denotation table, to get the meaning id. This increases the number of rows (for some languages, x5), because multiple meannings may be attached to the same expression.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

As with any multilingual dataset, Panlex data may exhbit the problem of under- and mis-representation of some languages.

The dataset consists primarily of the standard written forms ("lemmas") of the expressions, so it may not well represent their usage within a language.

Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

Kamholz, David, Jonathan Pool, and Susan M. Colowick. 2014. PanLex: Building a Resource for Panlingual Lexical Translation. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014).

BibTeX:

@inproceedings{kamholz-etal-2014-panlex,
    title = "{P}an{L}ex: Building a Resource for Panlingual Lexical Translation",
    author = "Kamholz, David  and
      Pool, Jonathan  and
      Colowick, Susan",
    editor = "Calzolari, Nicoletta  and
      Choukri, Khalid  and
      Declerck, Thierry  and
      Loftsson, Hrafn  and
      Maegaard, Bente  and
      Mariani, Joseph  and
      Moreno, Asuncion  and
      Odijk, Jan  and
      Piperidis, Stelios",
    booktitle = "Proceedings of the Ninth International Conference on Language Resources and Evaluation ({LREC}'14)",
    month = may,
    year = "2014",
    address = "Reykjavik, Iceland",
    publisher = "European Language Resources Association (ELRA)",
    url = "http://www.lrec-conf.org/proceedings/lrec2014/pdf/1029_Paper.pdf",
    pages = "3145--3150",
    abstract = "PanLex, a project of The Long Now Foundation, aims to enable the translation of lexemes among all human languages in the world. By focusing on lexemic translations, rather than grammatical or corpus data, it achieves broader lexical and language coverage than related projects. The PanLex database currently documents 20 million lexemes in about 9,000 language varieties, with 1.1 billion pairwise translations. The project primarily engages in content procurement, while encouraging outside use of its data for research and development. Its data acquisition strategy emphasizes broad, high-quality lexical and language coverage. The project plans to add data derived from 4,000 new sources to the database by the end of 2016. The dataset is publicly accessible via an HTTP API and monthly snapshots in CSV, JSON, and XML formats. Several online applications have been developed that query PanLex data. More broadly, the project aims to make a contribution to the preservation of global linguistic diversity.",
}

Glossary

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

To understand the terms like "language", "language variety", "expression" and "meaning" more precisely, please read the Panlex documentation on their data model and database design.