Datasetpublic

IntimateUser6969/text-classification-comparison

Text Classification Comparison: Supervised vs Unsupervised on stanfordnlp/imdb Dataset stanfordnlp/imdb (Maas et al., 2011) 50,000 IMDB movie reviews: 25,000 train / 25,000 test Binary sentiment: 0 (negative) / 1 (positive), perfectly balanced in both splits Preprocessing: TF-IDF (15,000 features, bigrams, sublinear TF, min_df=3) Models Chosen Model Type Key Reference Why Logistic Regression Linear supervised McFadden (1974); Ng &… See the full description on the dataset page: https://huggingface.co/datasets/IntimateUser6969/text-classification-comparison.

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes19downloads

Dataset Card

Text Classification Comparison: Supervised vs Unsupervised on stanfordnlp/imdb

Dataset

stanfordnlp/imdb (Maas et al., 2011)

—50,000 IMDB movie reviews: 25,000 train / 25,000 test
—Binary sentiment: 0 (negative) / 1 (positive), perfectly balanced in both splits
—Preprocessing: TF-IDF (15,000 features, bigrams, sublinear TF, min_df=3)

Models Chosen

Model	Type	Key Reference	Why
Logistic Regression	Linear supervised	McFadden (1974); Ng & Jordan (2000)	Optimal baseline for high-dimensional sparse text; widely benchmarked on TF-IDF
Random Forest	Ensemble (bagging)	Breiman (2001) "Random Forests"	Handles non-linearities via decorrelated tree averaging; strong baseline
XGBoost	Boosting	Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System"	Gradient boosting with L1/L2 regularization; won hundreds of Kaggle competitions
LightGBM	Boosting	Ke et al. (2017) "LightGBM: A Highly Efficient Gradient Boosting Decision Tree"	Leaf-wise growth + gradient-based sampling; faster for sparse features
KMeans	Unsupervised clustering	MacQueen (1967); Xie et al. (2021)	Standard unsupervised baseline with optimal label-mapping evaluation

Results (Test Set: 25,000 reviews)

Rank	Model	Accuracy	F1 (positive)	ROC-AUC	Train Time
1	Logistic Regression	0.8981	0.8987	0.9623	2.8s
2	XGBoost	0.8678	0.8695	0.9443	54.3s
3	LightGBM	0.8677	0.8698	0.9431	8.6s
4	Random Forest	0.8429	0.8478	0.9240	1.5s
5	KMeans (unsupervised)	0.4396	0.4728	0.4089	4.6s

Best Model: Logistic Regression (Accuracy=0.898, F1=0.899, AUC=0.962)

Why Logistic Regression Wins

1.Text classification is approximately linear: Sentiment in IMDB reviews is largely encoded in word-level cues (positive words like "great", "lovely" vs negative words like "bad", "poor"). TF-IDF features are highly sparse and high-dimensional (15,000 features). Regularized linear models are optimal in this regime.

1.No overfitting from high sparsity: The L2-regularized logistic regression finds a stable decision across the full vocabulary. Tree ensembles over-fit to rare word co-occurrences in sparse TF-IDF (most features are zero — ~99.7% of the 15,000 features are zero per sample).

1.Feature interactions captured by bigrams: TF-IDF includes unigrams and bigrams (ngram_range=(1,2)), which captures sentiment-carrying phrases ("not good", "very bad"). Logistic Regression leverages these directly as features, while tree models need many deep splits to approximate the same interactions.

1.Random Forest (0.84 acc) underperforms because:
2.Random feature subsampling (default sqrt(p) = 122 features per split) discards important unigram features
3.With high sparsity, most tree splits find no informative features at leaf nodes
4.Bagging doesn't help when individual trees are weak due to feature sparsity

1.XGBoost/LightGBM close second (0.87 acc, ~0.94 AUC) but require 54s vs 2.8s for LR

1.KMeans as expected: As an unsupervised approach, KMeans achieved 0.44 accuracy with ROC-AUC of 0.41 (below random). This confirms that without label supervision, word-frequency clustering cannot recover the sentiment signal.

References

1.Maas, et al. (2011). Learning Word Vectors for Sentiment Analysis. ACL.
2.Ng, Jordan (2000). On Discriminative vs Discriminative Learning. ICML.
3.Breiman (2001). Random Forests. Machine Learning.
4.Chen, Guestrin (2016). XGBoost. KDD.
5.Ke, et al. (2017). LightGBM. NeurIPS.
6.Vapnik (1995). The Nature of Statistical Learning Theory.
7.Xie, et al. (2021). A Survey of K-Means Clustering. arXiv:2112.06310.

Generated by ML Intern

This dataset repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

—Try ML Intern: https://smolagents-ml-intern.hf.space
—Source code: https://github.com/huggingface/ml-intern

Usage

python

from datasets import load_dataset

dataset = load_dataset('IntimateUser6969/text-classification-comparison')