Modelpublic

Hagrun/memerationerizer-faithfulness-entailer

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes15downloads

Model Card

Memerationerizer Faithfulness Entailer (sentence)

A 355M-parameter per-claim faithfulness entailer for verified memory compression. Given a source memory and a candidate compressed version, the model flags whether the candidate flips, drops, or adds facts relative to the source — running one sentence at a time with no external LLM call required at inference time. It is the verification backbone of the Memerationerizer system, which compresses AI agent memories with the guarantee that every fidelity-reduction step is fact-preserving. The coupling thesis is that decay must be gated by a faithfulness verifier: lossy compression of facts does not blur them — it flips them, and a flipped fact is wrong in a different way than a vague one.

How to use

python

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from mnemo.verify.minicheck import MiniCheckVerifier

# The verifier wraps the model in bidirectional unit-entailment:
# a candidate sentence is only "supported" if it both follows from the source
# AND the source implies it (no new facts, no dropped facts).
verifier = MiniCheckVerifier(
    model_name_or_path="Hagrun/memerationerizer-faithfulness-entailer",
    threshold=0.8,   # dev-selected; see Limitations for the --grid requirement
)

result = verifier.verify(source_memory, candidate_memory)
# result.verdict in {"faithful", "flipped", "dropped", "added"}

For the full harness — splitter, threshold-selection logic, and evaluation scripts — see the GitHub repository.

Training

—Base checkpoint: lytang/MiniCheck-RoBERTa-Large (355M, warm-started)
—Objective: binary cross-entropy (supported / unsupported)
—Teacher: claude-sonnet-4-6 unit-entailer — each (source, candidate) pair was split into sentence units and labeled bidirectionally by the teacher
—Corpus: train_sentence.jsonl — 1,413 rows, 1,068 supported (76%) / 345 unsupported (24%)
—Epochs / hardware: 3 epochs, Apple MPS (M4 Pro, 24 GB)
—Early stopping: 70-row dev set scored through the full MiniCheckVerifier pipeline; selection is lexicographic — maximize flip-recall first, then minimize false-alarm rate
—Dataset: Hagrun/memerationerizer-faithfulness-eval

Evaluation (held-out)

Dev-selected threshold τ = 0.8 (extended grid, see Limitations), applied to the held-out flips.jsonl ∪ hard.jsonl split:

Metric	This model (τ=0.8)	Sonnet ceiling	Phase 2a free-lever baseline
flip-recall	1.00	1.00	1.00
drop-recall	0.94	1.00	1.00
add-recall	1.00	1.00	1.00
false-alarm rate	0.00	0.00	0.11

The Phase 2a baseline (hard-FA 0.11) used only prompt-engineering on an off-the-shelf MiniCheck verifier. Training reduces false-alarms to 0.00 while holding flip-recall at 1.00 — the primary goal of Phase 2b.

Forgetting on LLM-AggreFact (full test split, 29,320 rows, τ=0.5 — the conventional MiniCheck operating point):

Model	macro balanced accuracy	Δ
off-the-shelf MiniCheck-RoBERTa-Large	0.7149	—
this model (sentence-trained)	0.6740	−0.041

Approximately 4 points of general balanced accuracy is traded for in-domain specialization.

Reproduce the held-out numbers:

bash

# Activate the project venv first (PyYAML and project deps required):
source .venv/bin/activate
python -m mnemo.eval.train_report --model-dir models/sentence --splitter sentence --grid 0.95,0.9,0.8,0.7,0.6,0.5,0.4,0.3,0.2,0.15,0.1,0.05 --name sentence-trained

Limitations

—One structural drop miss. hard.jsonl contains a single merged-sentence omission that the sentence splitter cannot surface as a separate unit. This leaves drop-recall at 0.92 on that file (0.94 on the combined held-out set), and the strict joint flip = drop = 1.00 condition is not met.

—τ = 0.8 requires the extended `--grid`. The default threshold grid caps at τ = 0.5. With the default grid, select_threshold picks τ = 0.5, which misses a flip on transfer (flip-recall drops to 0.96). Passing --grid 0.95,0.9,0.8,...,0.05 is required to reproduce the headline numbers; without it the default-grid result fails the flip guardrail. Sequencing note: the decision to extend the grid was prompted by a held-out diagnostic showing the flip guardrail holds at τ ≥ 0.8; dev independently selects τ = 0.8 once the grid exposes it (pure dev-argmax — held-out plays no part in threshold selection). The τ = 0.8 operating point is dev-selected, not eval-tuned, but this sequencing is disclosed so the result can be weighed honestly.

—~4 points of forgetting on general entailment. Specialization on a narrow distilled corpus trades measurable accuracy on out-of-domain datasets. In-domain use is unaffected; long-document AggreFact subsets are additionally impacted by the 512-token truncation limit.

—Lexical-only leakage check. The training / evaluation disjointness check uses token-overlap (not embeddings), so near-paraphrase leakage is not fully ruled out.

—Teacher task-shift. The claude-sonnet-4-6 teacher was used as a binary unit-entailer, which is a different task than the validated pair-judge it is designed for. The student inherits any blind spots from that task shift.

—Small, domain-narrow training corpus. 1,413 rows across 10 domains (operations, finance, personal, research, product, health, legal, travel, education, engineering). Warm-starting from MiniCheck makes the corpus viable; scale is the primary lever if the model is revisited.

License and data note

The base checkpoint (lytang/MiniCheck-RoBERTa-Large) is MIT licensed, as is the underlying roberta-large. This derivative is released under the same MIT license.