Datasetpublic

lbourdois/panlex

PanLex January 1, 2024 version of PanLex Language Vocabulary with 24,650,274 rows covering 6,152 languages. Columns vocab: contains the text entry. 639-3: contains the ISO 639-3 languages tags to allow users to filter on the language(s) of their choice. 639-3_english_name: the English language name associated to the code ISO 639-3. var_code: contains a code to differentiate language variants. In practice, this is the code 639-3 + a number. If 000, it… See the full description on the dataset page: https://huggingface.co/datasets/lbourdois/panlex.

sourceHugging Facecc0-1.0updated 3y agoView on Hugging Face

8likes125downloads

Dataset Card

PanLex

January 1, 2024 version of PanLex Language Vocabulary with 24,650,274 rows covering 6,152 languages.

Columns

—vocab: contains the text entry.
—639-3: contains the ISO 639-3 languages tags to allow users to filter on the language(s) of their choice.
—639-3_english_name: the English language name associated to the code ISO 639-3.
—var_code: contains a code to differentiate language variants. In practice, this is the code 639-3 + a number. If 000, it corresponds to the classic 639-3, if the number is different, it corresponds to a variant (e.g. fra-006 corresponds to Québécois according to https://vocab.panlex.org/fra-006). This is a code created by Panlex, and therefore does not follow an international standard.
—english_name_var: the English language name associated to the var_code.

Source

Readers are invited to consult the https://panlex.org/source-list/ page, which summarizes all the sources.

Citation

Kamholz, David, Jonathan Pool, and Susan M. Colowick. 2014.

PanLex: Building a Resource for Panlingual Lexical Translation. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014). http://www.lrec-conf.org/proceedings/lrec2014/pdf/1029_Paper.pdf