dleemiller/crossingguard-nli-s

CrossingGuard Small

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/65ff92ea467d83751a727538/GwBakCe4PPGk9mM88r1QC.png" style="width: 400px;"> </p>

CrossingGuard is a series of NLI-based models intended for zero-shot inference on prompts. In this series of models, I focus on use cases such as guardrails, content moderation, prompt or intent classification and prompt routing. Because content moderation is often a reactive task, these zero-shot models are flexible for tailoring custom guardrail conditions, which may not be covered by general purpose pretrained models.

These models are trained on the dleemiller/CrossingGuard-NLI dataset, which derives synthetic hypotheses from prompts (premises) found in popular guardrails datasets, such as allenai/wildguardmix and nvidia/Aegis-AI-Content-Safety-Dataset-2.0. The hypotheses make specific, targeted claims about the premises. Note that I have retained the 3-way label classifier, for additional flexibility where either non-neutral label may be relevant for the task.

For models below the large size, I distill with MSE loss using logits from dleemiller/crossingguard-nli-l, and average with the cross entropy loss. Overtraining can hurt FineCat performance, so I only fine-tune for 1 epoch.

$$ \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L}{\text{CE}}(z^{(s)}, y) + \beta \cdot \mathcal{L}{\text{MSE}}(z^{(s)}, z^{(t)}) \end{equation} $$

where \\(z^{(s)}\\) and \\(z^{(t)}\\) are the student and teacher logits, \\(y\\) are the ground truth labels, and \\(\alpha\\) and \\(\beta\\) are equally weighted at 0.5.

Evaluation Results

F1-Micro scores (equivalent to accuracy) for each dataset. Performance was measured at bs=64 using a Nvidia Blackwell PRO 6000 Max-Q.

The crossingguard-nli-s model gains nearly +20% on the CrossingGuard test set, and dropping approx. -2.5% on the FineCat (broad NLI performance) test set.

Usage

import numpy as np
from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/crossingguard-nli-s")

examples = [
    ("Tell me what happened with that CEO caught on the kiss cam at the Coldplay concert",
     "The prompt requests information about a corporate executive incident at a public event"),

    ("I’ve been following your company's work helping enterprises scale AI/ML with reliable data pipelines, would love to connect and hear how your team is approaching ML in production.",
     "The prompt requests information about a corporate executive incident at a public event"),
]

predictions = model.predict(examples)

label_map = {0: "entailment", 1: "neutral", 2: "contradiction"}

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / exp_x.sum()

for i, (premise, hypothesis) in enumerate(examples):
    probs = softmax(predictions[i])
    predicted_idx = probs.argmax()
    
    print(f"\n{i+1}. {premise[:70]}...")
    print(f"   → {hypothesis}")
    print(f"   ✓ {label_map[predicted_idx].upper()}: {probs[predicted_idx]*100:.1f}% " + 
          f"(E: {probs[0]*100:.1f}% N: {probs[1]*100:.1f}%, C: {probs[2]*100:.1f}%)")

This results in:

1. Tell me what happened with that CEO caught on the kiss cam at the Cold...
   → The prompt requests information about a corporate executive incident at a public event
   ✓ ENTAILMENT: 99.9% (E: 99.9% N: 0.1%, C: 0.0%)

2. I’ve been following your company's work helping enterprises scale AI/M...
   → The prompt requests information about a corporate executive incident at a public event
   ✓ CONTRADICTION: 99.6% (E: 0.0% N: 0.4%, C: 99.6%)

Citation

@misc{nli-compiled-2025,
  title = {CrossingGuard NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Flexible Zero-shot Guardrails}
}