Avifenesh/episodic-ingestion-compiler-modernbert-large-mh-5000

episodic-ingestion-compiler ModernBERT-large multi-head (5000 steps)

Fine-tuned from answerdotai/ModernBERT-large for the ingestion-model-spec-v1 task in the episodic-rs project: given a blurb of text, emit a list of structured claim candidates conforming to the spec-v1 wire format (closed 21-predicate allowlist, 13-subject-type allowlist, predicate-specific objectvalue, character offset sourcespans, calibrated confidence, first-class abstention).

This checkpoint is the production-selected backbone of the bakeoff conducted 2026-05-09. See the project's docs/bakeoff-decision-2026-05-09.md for full rationale.

Architecture

The encoder pools [CLS] into four heads:

Not directly loadable via AutoModel.from_pretrained. Use the IngestionEncoder wrapper in scripts/train_ingestion_encoder.py of the source repo.

Training

Eval metrics (val split, 13,179 blurbs)

Loading

The checkpoint is a torch.save of {state_dict, backbone, args, step, eval}. Load via:

import torch
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import hf_hub_download
import torch.nn as nn

class IngestionEncoder(nn.Module):
    def __init__(self, backbone_name: str):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(backbone_name)
        h = self.backbone.config.hidden_size
        self.dropout = nn.Dropout(0.1)
        self.head_claim_present = nn.Linear(h, 1)
        self.head_predicate = nn.Linear(h, 21)
        self.head_subject_type = nn.Linear(h, 13)
        self.head_confidence = nn.Linear(h, 1)

    def forward(self, input_ids, attention_mask):
        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        pooled = self.dropout(out.last_hidden_state[:, 0])
        return {
            "claim_logit": self.head_claim_present(pooled).squeeze(-1),
            "predicate_logits": self.head_predicate(pooled),
            "subject_type_logits": self.head_subject_type(pooled),
            "confidence_pred": torch.sigmoid(self.head_confidence(pooled).squeeze(-1)),
        }

ckpt = torch.load(
    hf_hub_download("Avifenesh/episodic-ingestion-compiler-modernbert-large-mh-5000", "best.pt"),
    map_location="cpu", weights_only=False,
)
model = IngestionEncoder(ckpt["backbone"])
model.load_state_dict(ckpt["state_dict"])
model.eval()
tokenizer = AutoTokenizer.from_pretrained(ckpt["backbone"])

Allowlists (closed vocabularies enforced by the spec)

Predicates (21): validated_by, had_outcome, failed_because, worked_because, decided, blocked_by, has_next_action, has_status, has_goal, touched_file, ran_command, logged_event, has_constraint, has_open_question, has_quality_finding, reverted_file, deleted_file, created_file, committed, deployed, incident_observed.

Subject types (13): objective, command, file, pr, incident, policy, person, repo, team, service, document, ticket, thread.

Wrapper responsibility (not done by this model)

This model emits the information-bearing subset. A deterministic wrapper fills the literal fields:

• —status = "candidate" (always)
• —source_authority = "model_draft" (always)
• —extraction_method = "ingestion_model_v1" (always)
• —source_memory_ids from the caller's scope
• —class_hint derived from predicate
• —qualifiers = {} default
• —source_spans TBD — current encoder emits whole-blurb as the single span; a span-boundary head is planned

Bakeoff context

Competed against: ModernBERT-base-MH, DeBERTa-v3-large-MH, NuExtract-2.0-2B (QLoRA SFT), NuExtract-2.0-8B (QLoRA SFT), Qwen3.5-4B (QLoRA SFT). ModernBERT-large-MH won on every gate.