CoolFace
Datasetpublic

nltk-data-hub/words

NLTK Word Lists English word lists from NLTK, the New General Service List Project, and Bing Liu's Opinion Lexicon. Configs Config Words Schema License Source en 235,886 word NLTK (other) NLTK words corpus en-basic 850 word Public domain Ogden Basic English (1930) ngsl 2,809 word, rank, sfi, freq_per_million CC-BY-SA 4.0 New General Service List 1.2 toeic 1,250 word, rank, sfi, freq_per_million CC-BY-SA 4.0 TOEIC Service List 1.2 nawl 963 word… See the full description on the dataset page: https://huggingface.co/datasets/nltk-data-hub/words.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face
1likes135downloads
Dataset Card

NLTK Word Lists

English word lists from NLTK, the New General Service List Project, and Bing Liu's Opinion Lexicon.

Configs

ConfigWordsSchemaLicenseSource
en235,886wordNLTK (other)NLTK words corpus
en-basic850wordPublic domainOgden Basic English (1930)
ngsl2,809word, rank, sfi, freq_per_millionCC-BY-SA 4.0New General Service List 1.2
toeic1,250word, rank, sfi, freq_per_millionCC-BY-SA 4.0TOEIC Service List 1.2
nawl963word, rank, band, sfi, freq_per_millionCC-BY-SA 4.0New Academic Word List 1.2
bsl1,744word, rank, band, sfi, freq_per_millionCC-BY-SA 4.0Business Service List 1.2
opinion-positive2,006wordCC-BY 4.0Hu & Liu Opinion Lexicon
opinion-negative4,783wordCC-BY 4.0Hu & Liu Opinion Lexicon

See Also

These related word list datasets are also accessible via nltk.corpus.words.words():

DatasetContentsNLTK access
nltk-data-hub/dolch315 Dolch sight words, 8 POS configswords.words("dolch"), words.words("dolch-verbs"), …
nltk-data-hub/swadesh207 Swadesh concepts × 24 languageswords.words("swadesh-en"), words.words("swadesh-de"), …

Schemas

`en`, `en-basic`, `opinion-positive`, `opinion-negative` — word only

ColumnTypeDescription
wordstringThe word

`ngsl` and `toeic` — frequency metadata, no band

ColumnTypeDescription
wordstringHeadword / lemma
rankintFrequency rank (1 = most frequent)
sfifloatStandard Frequency Index
freq_per_millionfloatAdjusted frequency per million words

`nawl` and `bsl` — frequency metadata + pedagogical band

ColumnTypeDescription
wordstringHeadword / lemma
rankintFrequency rank within this list
bandintPedagogical band grouping (lower = more frequent)
sfifloatStandard Frequency Index
freq_per_millionfloatAdjusted frequency per million words

Usage

python
from datasets import load_dataset

ds = load_dataset("nltk-data-hub/words", "ngsl")
ds = load_dataset("nltk-data-hub/words", "nawl")
ds = load_dataset("nltk-data-hub/words", "opinion-positive")
ds = load_dataset("nltk-data-hub/words", "opinion-negative")

Via NLTK

python
import nltk
nltk.download("words", hf=True)

nltk.corpus.words.words("ngsl")              # 2,809 words, frequency order
nltk.corpus.words.words("nawl")              # 963 academic words
nltk.corpus.words.words("bsl")               # 1,744 business words
nltk.corpus.words.words("toeic")             # 1,250 TOEIC words
nltk.corpus.words.words("opinion-positive")  # 2,006 positive opinion words
nltk.corpus.words.words("opinion-negative")  # 4,783 negative opinion words
nltk.corpus.words.words("en")                # 235,886 words
nltk.corpus.words.words("en-basic")          # Ogden 850
# Routed to nltk-data-hub/dolch:
nltk.corpus.words.words("dolch")             # 315 Dolch sight words
nltk.corpus.words.words("dolch-verbs")       # 92 Dolch verbs
# Routed to nltk-data-hub/swadesh:
nltk.corpus.words.words("swadesh-en")        # 207 English Swadesh words
nltk.corpus.words.words("swadesh-de")        # 207 German Swadesh words

Licenses

  • —en, en-basic: distributed as part of the NLTK corpus data package.
  • —ngsl, toeic, nawl, bsl: © Browne, Culligan & Phillips, licensed under CC-BY-SA 4.0.
  • —opinion-positive, opinion-negative: © Bing Liu, licensed under CC-BY 4.0.

Citations

bibtex
@book{nltk,
  author    = {Bird, Steven and Klein, Ewan and Loper, Edward},
  title     = {Natural Language Processing with Python},
  publisher = {O'Reilly Media},
  year      = {2009},
  url       = {https://www.nltk.org/}
}

@article{ngsl,
  author    = {Browne, Charles},
  title     = {A New General Service List: The Better Mousetrap We've Been Looking For?},
  journal   = {Vocabulary Learning and Instruction},
  volume    = {3},
  number    = {2},
  pages     = {1--10},
  year      = {2014},
  doi       = {10.7820/vli.v03.2.browne}
}

@misc{nawl,
  author    = {Browne, Charles and Culligan, Brent and Phillips, Joseph},
  title     = {New Academic Word List 1.2},
  year      = {2013},
  url       = {https://www.newgeneralservicelist.com/nawl-new-academic-word-list}
}

@misc{tsl,
  author    = {Browne, Charles and Culligan, Brent},
  title     = {TOEIC Service List 1.2},
  year      = {2016},
  url       = {https://www.newgeneralservicelist.com/toeic-service-list}
}

@misc{bsl,
  author    = {Browne, Charles and Culligan, Brent},
  title     = {Business Service List 1.2},
  year      = {2016},
  url       = {https://www.newgeneralservicelist.com/business-service-list}
}

@inproceedings{opinion_lexicon,
  author    = {Hu, Minqing and Liu, Bing},
  title     = {Mining and Summarizing Customer Reviews},
  booktitle = {Proceedings of KDD-2004},
  year      = {2004},
  url       = {http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html}
}