Modelpublic

HandongAILab/monli-bert-snli

sourceHugging Facecc-by-sa-4.0updated 4mo agoView on Hugging Face

0likes9downloads

Model Card

monli-bert-snli

bert-base-uncased fine-tuned on SNLI for 3-way Natural Language Inference (entailment / neutral / contradiction).

This is a reproduction artifact — the Experiment 1.1 baseline from our reproduction of:

Atticus Geiger, Kyle Richardson, Christopher Potts (2020). Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation. BlackboxNLP @ EMNLP 2020. arXiv:2004.14623

It is not a novel model; it exists to evaluate how a standard SNLI-trained NLI model behaves on the MoNLI monotonicity challenge set.

Training

Setting	Value
Base model	`bert-base-uncased`
Data	SNLI train (~549K examples; rows with `label == -1` removed)
Epochs	3
Learning rate	2e-5
Batch size	48
Max length	128
Warmup ratio	0.1

Results

Evaluated on SNLI test and on MoNLI (Geiger et al. paper figures in parentheses):

Dataset	Accuracy
SNLI test	90.6% (paper 90.8%)
PMoNLI (no negation)	90.7% (paper 94.4%)
NMoNLI (with negation)	2.7% (paper 2.2%)

The core finding is reproduced: high accuracy without negation, near-0% once negation reverses the entailment direction — SNLI training provides essentially no signal for negation + monotonicity.

Labels

id	label
0	entailment
1	neutral
2	contradiction

Reproduction

Companion model (after inoculation): `HandongAILab/monli-bert-inoculated`

Limitations

Trained only on SNLI; it does not handle negation-driven monotonicity and should not be used for production NLI. Inherits SNLI's known annotation artifacts.