Modelpublic

Avifenesh/episodic-ingestion-modernbert-field-event-ranker-mixed-v2-h4-320

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes8downloads

Model Card

episodic-ingestion-compiler / ModernBERT field-event ranker (mixed-mode v2, H4-320)

Fine-tune of answerdotai/ModernBERT-base for the grouped multi-positive softmax ranker task, trained on v2 semantic-reasoning labels for 320 optimizer steps (vs 160 in the initial v2 checkpoint). The extra training let the model fully converge on both legacy and semantic fields.

Versus v2 at 160 steps

Metric	v2-H4 (160)	v2-H4 (320)	Δ
overall MRR	0.526	0.583	+0.057
top-1	0.296	0.375	+0.079
last loss	0.0854	0.0000 (converged)

Training details

—Train rows: 1217 (mixed-mode v2)
—Train groups: 5882
—Eval rows: 220 (21986 candidate pairs)
—Steps: 320 optimizer steps × accum=8 = 2560 forwards
—Optimizer: AdamW lr 5.7e-5, warmup 30
—Precision: bf16 autocast + gradient checkpointing
—Peak VRAM: 4.08 GiB
—Hardware: RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB)

Eval metrics

metric	value
overall MRR	0.583
top-1 recall	0.375
top-2 recall	0.580
top-3 recall	0.737
top-5 recall	0.909
mean expected rank	2.67

Per-field MRR (top fields in eval):

field	n	MRR	top-1	random	ratio
`attempted_actions`	846	0.480	0.24	0.253	1.90x
`observed_outcomes`	845	0.491	0.24	0.253	1.94x
`initiating_command`	199	0.966	0.94	0.275	3.51x
`outcome_of_latest_attempt`	181	0.966	0.94	0.276	3.50x
`action_causality`	141	0.561	0.30	0.254	2.21x
`attempt_outcome_pairs`	53	0.616	0.34	0.265	2.33x
`failed_attempts`	36	0.705	0.53	0.256	2.75x
`next_actions`	32	0.755	0.56	0.296	2.55x
`recent_error`	29	0.826	0.72	0.256	3.23x
`customer_identity`	11	0.811	0.73	-	-
`transaction_reference`	5	0.633	0.40	-	-
`product_name`	4	1.000	1.00	-	-
`discarded_options`	2	0.375	0.00	-	-
`invalidation_hints`	2	0.750	0.50	-	-
`non_promotable_context`	2	0.417	0.00	-	-
`payment_or_warranty_detail`	2	0.333	0.00	-	-
`assistant_claims_to_verify`	1	0.500	0.00	-	-
`explicit_decisions`	1	1.000	1.00	-	-
`resolved_context`	1	0.333	0.00	-	-
`touched_files`	1	0.200	0.00	-	-
`unsupported_hypotheses`	1	1.000	1.00	-	-

Lineage

checkpoint	labels	steps	MRR	top-1	notes
V2 (commit 87cb089)	conversational	160	0.678	0.440	8-row adversarial eval only
mixed-mode stage-4	v1	160	0.323	0.130	pre-perf stack
mixed-mode perf-H4	v1	160	0.506	0.260	perf-H4 stack but v1 (role-tautological) labels
mixed-mode v2-H4	v2	160	0.526	0.296	first v2 train — non-converged
mixed-mode v2-H4-320 (this)	v2	320	0.583	0.375	converged — strongest checkpoint

Semantic-reasoning fields show clear learning

The fields designed to NOT be role-tautological reached high MRR through training, confirming the model is actually reasoning beyond role:

—initiating_command: MRR 0.966 (3.51x random) — learns trace boundaries
—outcome_of_latest_attempt: MRR 0.966 (3.50x random) — matches call_id chains
—recent_error: MRR 0.826 (3.23x random) — latest-failure reasoning
—action_causality: MRR 0.561 (2.21x random) — content grounding across turns

Legacy fields (attempted_actions, observed_outcomes) sit near 0.48, close to random for multi-positive softmax. They're role-tautological on 97% of rows, so the model can hit this with role-inference alone.

Intended use

Ranker head of a multi-stage episodic-ingestion compiler. Input is a JSON-serialized (requested field, candidate event) record. Output is a scalar logit that, after grouped softmax over siblings in the same trace, estimates the probability that the candidate event supports the field.

See docs/ranker-hypothesis-log-2026-05-08.md in the repo for the full experimental ladder (H1-H3b-span falsified, label-pivot kept, H3b-replay-v2 falsified, v2-H4-320 kept).