chcaa/dagw-word-frequencies-by-domain

Dataset Card for DAGW Word Frequencies (by domain)

• —Paper: Derczynski, L., Ciosici, M. R., Baglini, R., Christiansen, M. H., Dalsgaard, J. A., Fusaroli, R., ... & Varab, D. (2021). The Danish Gigaword Corpus. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa) (pp. 413-421).
• —Point of Contact: Kenneth Enevoldsen (Kennethcenevoldsen (at) gmail (dot) com )

This is a list of word frequencies derived from the Danish Gigaword (collected before 2022-22-01). These word frequencies are derived from tokens from the Danish Gigaword Corpus, which have been tokenized using the spacy pipeline for Danish "da_core_news_lg" using spacy>=3.0.0,<3.4.0. See the notebook "convert_to_hf_dataset.ipynb" and wordfreq.py for more information about how it was created.

Dataset formats

This dataset have been created in four formats:

• —`chcaa/dagw-word-frequencies`: Danish word frequencies from Danish Gigaword.
• —`chcaa/dagw-word-frequencies-by-domain`: word frequencies pr. domain.
• —`chcaa/dagw-word-frequencies-by-domain-with-pos-tags`: word frequencies pr. domain with their part-of-speech tags derived from the spacy pipeline for Danish "da_core_news_lg".
• —`chcaa/dagw-word-frequencies-normalized-by-domain`: word frequencies pr. domain normalized by the top-level domain.

Dataset Creation

Curation Rationale

Word frequencies of large domains har often used to calculate metrics such as text entropy or word surprise. Word frequencies can also be used to to create stopword lists and similar.

Source Data

The frequencies are derived from the Danish Gigaword. To read more about the Danish Gigaword and its content please check out the entry on Danish language resources, which also links to latest publications.

Discussion of Biases

This dataset contains notably different distributions of the original domains; for instance, the legal domain is highly overrepresented within this corpus. Please see the normalized version of this dataset if you wish to see word frequencies which normalized across domains.

Other Known Limitations

The news data within this corpus ("danavis") have altered the text, this is described further in the original version (v1) of paper.

Licensing Information

Below follows a brief overview of the sources in the corpus along with their individual license.

Contributions

Thanks to @KennethEnevoldsen for adding this dataset.