CCB/cis5300-text-classification

Complex Word Identification (CIS 5300)

Dataset Description

This dataset supports the Complex Word Identification (CWI) task: given a word in context, predict whether it is complex (likely to be difficult for non-native speakers, children, or people with reading disabilities) or simple.

CWI is the first step in lexical simplification — the task of rewriting text to make it more accessible. Before you can simplify a word, you need to identify which words need simplification.

Task

• —Input: A word and the sentence it appears in
• —Output: Binary label — 0 (simple) or 1 (complex)
• —Example: In the sentence "The beleaguered director resigned Wednesday", the word beleaguered is complex (label=1) while director is simple (label=0).

Dataset Structure

Default Configuration (Main Task)

from datasets import load_dataset

dataset = load_dataset("CCB/cis5300-text-classification")

# Splits: train (4,000), validation (1,000), test (922)
print(dataset["train"][0])
# {'word': 'string', 'label': 0, 'annotators': 0, 'sentence': 'WASHINGTON -- The beleaguered...', 'sentence_index': 27}

Fields:

Domain Generalization Configs

Two additional test sets from different domains for evaluating how well models generalize:

# Biomedical domain (PubMed abstracts)
bio = load_dataset("CCB/cis5300-text-classification", "biomedical")

# News domain (from CWI 2018 shared task)
news = load_dataset("CCB/cis5300-text-classification", "news")

Supplementary Data

The file ngram_counts.txt.gz contains word frequency counts from Google Books N-grams (~8.7M entries). This is useful as a feature for classification but is not a structured dataset. Download it separately:

from huggingface_hub import hf_hub_download
path = hf_hub_download("CCB/cis5300-text-classification", "ngram_counts.txt.gz", repo_type="dataset")

import gzip
ngram_counts = {}
with gzip.open(path, 'rt', encoding='utf-8') as f:
    for line in f:
        parts = line.strip().split('\t')
        if len(parts) == 2:
            ngram_counts[parts[0]] = int(parts[1])

Source

The core dataset was introduced in:

Simplification Using Paraphrases and Context-Based Lexical Substitution Reno Kriz, Eleni Miltsakaki, Marianna Apidianaki, Chris Callison-Burch Proceedings of NAACL-HLT 2018, pages 207–217 https://aclanthology.org/N18-1019/

The words are drawn from news articles and annotated by both native and non-native English speakers who indicated which words could be difficult for non-native speakers, children, or people with reading disabilities.

Annotation scheme and binarization

Each word was independently labeled by approximately 10 annotators. The annotators column records how many annotators marked the word as complex. Binary labels were derived using a threshold:

• —Simple (0): No annotator marked the word as complex (annotators = 0)
• —Complex (1): Three or more annotators marked it as complex (annotators >= 3)
• —Excluded: Words in the ambiguous zone (1–2 annotators) were removed from the dataset

This means the dataset contains only clear cases — words that are unambiguously simple or where a meaningful fraction of annotators agreed on complexity.

Domain generalization sources

• —Biomedical: From the CompLex dataset (Shardlow et al., 2020) — words from biomedical abstracts with continuous complexity scores binarized at threshold 0.5.
• —News: From the CWI 2018 Shared Task (Yimam et al., 2018) — words from news articles annotated for complexity.

Intended Use

This dataset is used for Homework 1 in CIS 5300: Natural Language Processing at the University of Pennsylvania. Students build progressively more sophisticated classifiers:

1. 1.Baselines: Word length and word frequency thresholds
2. 2.Machine learning: Naive Bayes and Logistic Regression with engineered features
3. 3.Custom models: Additional features with error analysis
4. 4.Domain generalization: Evaluate on biomedical and news text

Citation

@inproceedings{kriz-etal-2018-simplification,
    title = "Simplification Using Paraphrases and Context-Based Lexical Substitution",
    author = "Kriz, Reno and Miltsakaki, Eleni and Apidianaki, Marianna and Callison-Burch, Chris",
    booktitle = "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    year = "2018",
    address = "New Orleans, Louisiana",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/N18-1019",
    doi = "10.18653/v1/N18-1019",
    pages = "207--217",
}