joshdavham/multilingual-frequency-lists
Multilingual Frequency Lists This dataset contains multiple word-frequency lists in various languages such as French, Japanese, Spanish, Italian and Portuguese. Specifically, these are frequency lists of lemmas, meaning, for example, that words like 'run', 'runs' and 'running' are counted together as occurences of the same lemma 'run'. These frequency lists were generated from ~1GB of subtitles scraped from a variety of Netflix shows and films and parsed using relevant spacy… See the full description on the dataset page: https://huggingface.co/datasets/joshdavham/multilingual-frequency-lists.
Multilingual Frequency Lists
This dataset contains multiple word-frequency lists in various languages such as French, Japanese, Spanish, Italian and Portuguese. Specifically, these are frequency lists of lemmas, meaning, for example, that words like 'run', 'runs' and 'running' are counted together as occurences of the same lemma 'run'.
These frequency lists were generated from ~1GB of subtitles scraped from a variety of Netflix shows and films and parsed using relevant spacy models for each language. Each frequency list contains between 15k and 20k of the top most frequent words.
Quickstart
from datasets import load_dataset
from pprint import pprint
language = "ja" # japanese
dataset = load_dataset("joshdavham/multilingual-frequency-lists", language)
pprint(dataset["train"][:5])
# > {'count': [181501, 97107, 88598, 81836, 70747],
# 'lemma': ['ない', 'です', 'てる', 'ます', 'から'],
# 'rank': [1, 2, 3, 4, 5]}License
This dataset is licensed under the AGPL.