navihat/peer-review-claim-relation-classifier

ReviewSynth NLI — Relation Classifier for Peer Review Claims

Fine-tuned 6-class relation classifier that decides how two atomic claims from different peer reviewers of the same paper relate to each other — e.g. do two reviewers agree, partially agree, flag different symptoms of the same issue, or contradict each other outright.

Base model: `MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7` (mDeBERTa-v3-base, ≈0.3B params), fine-tuned on 989 hand/LLM-labeled claim pairs.

Full training code, data pipeline, and the FastAPI inference server live at [github.com/navihat/peer-review-claim-relations](https://github.com/navihat/peer-review-claim-relations).

Label space

AGREEMENT — PARTIAL_AGREEMENT — COMPLEMENTARY — PARTIAL_CONTRADICTION — CONTRADICTION
                                      ⊥
                                  UNRELATED

The full decision tree used to label the data is in `RUBRIC_GOLD.md` in the GitHub repo.

How to use

This model classifies a pair of claims, not a single text — that's why the Hub's default inference widget is disabled above (it only supports single-text input, and it can't reproduce the symmetric test-time augmentation this model needs for good accuracy). Use the snippet below instead.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

LABELS = ["AGREEMENT", "PARTIAL_AGREEMENT", "COMPLEMENTARY",
          "PARTIAL_CONTRADICTION", "CONTRADICTION", "UNRELATED"]

tok   = AutoTokenizer.from_pretrained("navihat/reviewsynth-nli")
model = AutoModelForSequenceClassification.from_pretrained(
            "navihat/reviewsynth-nli").eval()

def predict(left: str, right: str) -> str:
    """Symmetric TTA: average logits over both orderings before argmax."""
    with torch.no_grad():
        logits = sum(
            model(**tok(a, b, return_tensors="pt",
                        truncation=True, max_length=160)).logits
            for a, b in [(left, right), (right, left)]
        ) / 2
    return LABELS[logits.argmax().item()]

print(predict(
    "The method is well-motivated and addresses a real gap.",
    "I fail to see why this approach is needed over prior work."
))
# → CONTRADICTION

Always pass both orderings and average the logits. Skipping symmetric TTA degrades accuracy on order-sensitive examples — order-sensitivity was the root failure mode of the LLM ensemble originally used to build this dataset (flip-rate 44-50% raw vs. ~10% for this fine-tuned model).

A ready-to-run FastAPI server (/predict and a batch /v1/relations:predict endpoint) is in the GitHub repo.

Training data

989 claim pairs extracted from ICLR / NeurIPS peer reviews and labeled with Claude using a fixed rubric.

Split by paper group to prevent claim leakage between train and eval (train 788 / val 100 / test 101 pairs, 177/15/18 papers respectively).

Label distribution in train:

The high UNRELATED share reflects how the pairs were mined (same paper, same broad aspect → many claims that share a topic but not a specific issue).

Performance

All silver labels are Claude-generated. Numbers measure agreement with those labels, not absolute ground truth.

The gold test set contains 129 human-verified pairs from a different domain and language (Vietnamese university grant reviews). The cross-lingual / cross-domain drop from the English test split to this gold set is only ~0.04, which indicates strong multilingual transfer from the mDeBERTa-v3 backbone.

Flip-rate (how often the model changes its prediction when the two claims are swapped) on raw logits: ~10%, down from 44–50% for the raw LLM labelers used during data construction.

Per-class F1 (5-fold CV, silver):

† UNRELATED F1 is suppressed by prior mismatch (see Limitations below). CONTRADICTION has only 25 train / 2 test samples — read its F1 from CV, not from the test split.

Design decisions

Symmetric training + inference. The relation between claim A and claim B must not depend on which is listed first. Three mechanisms enforce this:

Class weights. CrossEntropyLoss(weight=inverse_frequency) so the model does not ignore rare classes. CONTRADICTION receives ~5–6× weight.

Paper-grouped split. Five folds stratified by label, groups defined at the paper level.

Limitations

• —Labels are LLM-generated. No independent human annotator on the training set → no human agreement ceiling. A macro-F1 of 0.25 cannot be interpreted without knowing inter-annotator agreement on the same task.
• —CONTRADICTION is rare (25/989 train samples). F1 on the 2-sample test split is not informative; use the 5-fold CV numbers.
• —Prior mismatch on deployment data. UNRELATED is 35% of training data but only 14% of the gold test set. If your deployment distribution differs substantially, consider a prior correction on the logits.
• —Training domain: English ML conference reviews. Transfer to other peer-review venues or languages will vary; the gold test (Vietnamese grant reviews, 0.25 macro-F1) is a single data point, not a guarantee.
• —No second annotator. The only human annotation is the 129-pair gold set. All 989 training labels are LLM-generated.

Citation

@misc{reviewsynth-nli-2026,
  author = {Truong Van Thai},
  title  = {ReviewSynth NLI: 6-class Relation Classifier for Peer Review Claims},
  year   = {2026},
  url    = {https://huggingface.co/navihat/reviewsynth-nli}
}