lbourdois/panlex

PanLex January 1, 2024 version of PanLex Language Vocabulary with 24,650,274 rows covering 6,152 languages. Columns vocab: contains the text entry. 639-3: contains the ISO 639-3 languages tags to allow users to filter on the language(s) of their choice. 639-3_english_name: the English language name associated to the code ISO 639-3. var_code: contains a code to differentiate language variants. In practice, this is the code 639-3 + a number. If 000, it… See the full description on the dataset page: https://huggingface.co/datasets/lbourdois/panlex.