gal-lardo/BERT-RTE-LinearClassifier-v2

BERT-RTE Linear Classifier for EEE 486/586 Assignment

This model is a fine-tuned version of bert-base-uncased on the RTE (Recognizing Textual Entailment) task from the GLUE benchmark. It was developed as part of the EEE 486/586 Statistical Foundations of Natural Language Processing course assignment.

Model Architecture

Unlike the standard BERT classification approach, this model implements a custom architecture:

• —Uses BERT base model as the encoder for feature extraction
• —Replaces the standard single linear classification head with multiple linear layers:
• —First expansion layer: hiddensize → hiddensize
• —Intermediate layer with ReLU activation and dropout
• —Final classification layer
• —Uses label smoothing of 0.1 in the loss function for better generalization

Performance

The model achieves 69.31% accuracy on the RTE validation set, with the following training dynamics:

• —Best validation accuracy: 69.31% (epoch 4)
• —Final validation accuracy: 69.31% (with early stopping)

Hyperparameters

The model was optimized using Optuna hyperparameter search:

Usage

This model can be used for textual entailment classification (determining whether one text logically follows from another):

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("gal-lardo/BERT-RTE-LinearClassifier")
model = AutoModelForSequenceClassification.from_pretrained("gal-lardo/BERT-RTE-LinearClassifier")

# Prepare input texts
premise = "The woman is sleeping on the couch."
hypothesis = "There is a woman resting."

# Tokenize and predict
inputs = tokenizer(premise, hypothesis, return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs)
prediction = outputs.logits.argmax(-1).item()

# Convert prediction to label
label = "entailment" if prediction == 1 else "not_entailment"
print(f"Prediction: {label}")