AmjaadXX/sentiment-analysis-distilbert-rotten-tomatoes

license: mit datasets:

• —rotten_tomatoes language:
• —en metrics:
• —none base_model:
• —distilbert-base-uncased pipelinetag: text-classification libraryname: transformers tags:
• —nlp
• —sentiment-analysis
• —text-classification
• —distilbert
• —data-preprocessing ---

Sentiment Analysis NLP Pipeline (Rotten Tomatoes)

This project builds a complete NLP data processing pipeline for sentiment analysis using the Rotten Tomatoes dataset.

The workflow focuses on:

• —Data exploration
• —Data cleaning
• —Feature engineering
• —Tokenization and model preparation

No model training has been performed yet. This project prepares the dataset for training transformer-based models such as DistilBERT.

Dataset

We use the Rotten Tomatoes dataset from Hugging Face Datasets library. It contains movie reviews labeled as:

• —0 → Negative
• —1 → Positive

Project Steps

1. Data Loading

The dataset is loaded using the Hugging Face datasets library and inspected for structure, splits, and features.

2. Data Exploration

The dataset was analyzed to:

• —View sample examples
• —Inspect schema (features)
• —Check label distribution across splits

3. Label Processing

Numeric labels were mapped to text labels:

• —0 → negative
• —1 → positive

A new column label_text was added.

4. Train/Test Split

The training data was split into:

• —75% training
• —25% testing

This ensures reproducibility using a fixed random seed.

5. Data Cleaning

Text data was cleaned using:

• —Lowercasing
• —Removing special characters using regex
• —Removing extra whitespace

Additional features were created:

• —clean_text
• —char_length
• —word_count

6. Data Filtering

Very short texts (less than 3 words) were removed to improve data quality.

7. Tokenization

The dataset was tokenized using:

• —distilbert-base-uncased tokenizer
• —max_length = 96
• —padding = max_length
• —truncation enabled

8. Model Preparation

The dataset was formatted for PyTorch:

• —input_ids
• —attention_mask
• —labels

Unused columns were removed to prepare for training.

9. Dataset Subsets

For experimentation:

• —300 training samples were selected
• —100 test samples were selected

Output Summary

The final dataset is fully prepared for training transformer-based models using Hugging Face Trainer or PyTorch.

Model Status

No model training or evaluation has been performed yet. This project focuses on building a clean and structured NLP preprocessing pipeline.

Tools Used

• —Hugging Face Datasets
• —Hugging Face Transformers
• —PyTorch
• —Pandas
• —Regex

Future Work

• —Fine-tune DistilBERT on the prepared dataset
• —Add evaluation metrics (accuracy, F1-score)
• —Deploy model using Hugging Face Spaces