ErfanMoosaviMonazzah/backpack-gpt2-nli
121
Model Card for Backpack-GPT2-NLI
This is a fine-tuned version of backpack-gpt2 with a NLI classification head on the esnli dataset. Results:
- On Validation Set:
- CrossEntropyLoss: 0.3168
- Accuracy: 0.9006
- F1: 0.9004
- On Test Set:
- CrossEntropyLoss: 0.3277
- Accuracy: 0.8958
- F1: 0.8955
Model Description
- Developed by: Erfan Moosavi Monazzah
- Model type: Sequence Classifier
- Language(s) (NLP): English
- License: apache-2.0
- Finetuned from model [optional]: Backpack-GPT2
How to Get Started with the Model
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token
def tokenize_function(examples):
concatenated_sentences = [f'{premise.strip(".")}. ^ {hypothesis.strip(".")}.' for premise, hypothesis in zip(examples['premise'], examples['hypothesis'])]
tokenized_inputs = tokenizer(
concatenated_sentences,
padding="max_length",
truncation=True,
max_length=512,
return_tensors="pt",
)
return tokenized_inputs
model = AutoModelForSequenceClassification.from_pretrained('ErfanMoosaviMonazzah/backpack-gpt2-nli', trust_remote_code=True)
model.eval()
tokenized_sent = tokenize_function({
'premise':['A boy is jumping on skateboard in the middle of a red bridge.',
'Two women who just had lunch hugging and saying goodbye.',
'Children smiling and waving at camera'],
'hypothesis':['The boy does a skateboarding trick.',
'The friends have just met for the first time in 20 years, and have had a great time catching up.',
'The kids are frowning']
})
model.predict(input_ids=tokenized_sent['input_ids'], attention_mask=tokenized_sent['attention_mask'])Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-5
- trainbatchsize: 64
- evalbatchsize: 64
- seed: 2023
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lrschedulertype: linear
- lrschedulerwarmup_ratio: 0
- num_epochs: 3
