rishhh/verisci-claim-verifier-sentence-nli

VeriSci Claim Verifier

NLI-pretrained DeBERTa v3 base sequence classifier for evidence-aware scientific claim verification over SUPPORTS, REFUTES, and NOT_ENOUGH_INFO.

Intended Use

Given a scientific claim and retrieved evidence passage, classify whether the evidence supports, refutes, or does not provide enough information. This model is not a medical device, not a substitute for peer review, and should not be used for clinical, legal, or public-policy decisions without expert review.

Training Data

• —Primary verifier data: allenai/scifact_entailment train split.
• —Hard NEI examples: andreiaalexa/scifact-relevance-pairs title/train not_relevant pairs.
• —Validation/test policy: train-overlapping validation claims removed, then remaining validation rows partitioned by normalized claim key. Derived test was not used for training or model selection.

Evaluation

Validation accuracy: 0.8452 Validation macro F1: 0.8269

Derived test accuracy: 0.8876 Derived test macro F1: 0.8503

See evaluation/eval_summary.json and prediction JSONL files for class metrics, confusion matrices, and limitations.

Limitations

• —The derived test split is small and originates from SciFact validation, not an official hidden test set.
• —The model only judges the provided evidence; retrieval failures can change the end-to-end answer.
• —Raw softmax confidence is not a calibrated probability.
• —Training data is SciFact-derived and subject to license/domain constraints.