Modelpublic

Avifenesh/episodic-ingestion-modernbert-field-event-ranker-mixed-v2-perf-h4

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes10downloads

Model Card

episodic-ingestion-compiler / ModernBERT field-event ranker (mixed-mode v2, perf-H4)

Fine-tune of answerdotai/ModernBERT-base for the grouped multi-positive softmax ranker task with v2 semantic-reasoning labels.

What changed vs the v1 mixed-mode checkpoint

An audit of the v1 mixed-mode training data showed that 94% of eval pairs came from two fields (attempted_actions, observed_outcomes) whose labels were exact functions of event role on 97% of rows (attemptedactions=toolcall, observedoutcomes=toolresult). On these degenerate labels:

—E[MRR | random scores] ≈ 0.26 (random-permutation floor for multi-positive softmax)
—v1 perf-H4 MRR = 0.506 ≈ 1.97× random
—Architecture experiments (H3, H3a, H3b, H3b-span, etc.) could not break through because the ceiling was arithmetic, not informational.

v2 adds five semantic-reasoning fields whose labels require more than role:

field	kind	rule
`initiating_command`	single-pick	First tool_call in a multi-call chain
`outcome_of_latest_attempt`	single-pick	Toolresult whose callid matches the LAST tool_call
`recent_error`	single-pick	Most-recent tool_result with success=false
`attempt_outcome_pairs`	array	Paired toolresults (only when window has ≥1 orphan result — otherwise empty, so it can't collapse to role==toolresult)
`action_causality`	array	Tool_results whose content tokens recur in a LATER assistant turn

Tautology rates on these new fields: 0–10%. Legacy fields kept for back-compat.

Training details

—Train rows: 1217 (mixed-mode v2: autonomous + prompt→action + customer-support + conversational)
—Train groups: 5882 (one per field × row pair)
—Eval rows: 220 (1900 groups, 21986 candidate pairs)
—Steps: 160 optimizer steps (--accum-steps 8)
—Optimizer: AdamW, lr 5.7e-5, warmup 30
—Precision: bf16 autocast
—Memory: gradient checkpointing (use_reentrant=False)
—Final train loss: 0.0854
—Peak VRAM: 4.08 GiB
—Hardware: RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB)

Eval metrics (220-row mixed v2 eval, {'1': 0.2964509394572025, '2': 0.5173277661795407, '3': 0.6931106471816284, '5': 0.8881002087682672})

metric	value
overall MRR	0.526
top-1 recall	0.296
top-2 recall	0.517
top-3 recall	0.693
top-5 recall	0.888
mean expected rank	3.09

Per-field MRR (sorted by eval-pair count):

field	n	MRR	top-1
`attempted_actions`	846	0.469	0.23
`observed_outcomes`	845	0.486	0.24
`initiating_command`	199	0.667	0.50
`outcome_of_latest_attempt`	181	0.763	0.60
`action_causality`	141	0.483	0.24
`attempt_outcome_pairs`	53	0.563	0.30
`failed_attempts`	36	0.615	0.42
`next_actions`	32	0.565	0.34
`recent_error`	29	0.743	0.59
`customer_identity`	11	0.818	0.73
`transaction_reference`	5	0.467	0.20
`product_name`	4	1.000	1.00
`discarded_options`	2	0.375	0.00
`invalidation_hints`	2	0.750	0.50
`non_promotable_context`	2	0.333	0.00
`payment_or_warranty_detail`	2	0.350	0.00
`assistant_claims_to_verify`	1	0.500	0.00
`explicit_decisions`	1	1.000	1.00
`resolved_context`	1	0.250	0.00
`touched_files`	1	0.333	0.00
`unsupported_hypotheses`	1	0.333	0.00

Lineage

checkpoint	labels	overall MRR	top-1	notes
V2 (commit 87cb089)	conversational only	0.678	0.440	8-row adversarial eval, not comparable
mixed-mode stage-4	v1	0.323	0.130	pre-perf stack
mixed-mode perf-H4	v1	0.506	0.260	winning perf stack — but on role-tautological labels
mixed-mode v2 perf-H4 (this)	v2	0.526	0.296	same perf stack, semantic-reasoning labels

Intended use

Ranker head of a multi-stage episodic-ingestion compiler. Input is a JSON-serialized (requested field, candidate event) record. Output is a scalar logit that, after grouped softmax over siblings in the same trace, estimates the probability that the candidate event supports the requested field.

Limitations

—Aggregate MRR (0.526) is a blend across fields of very different difficulty. Legacy fields (attempted_actions, observed_outcomes) are still role-tautological on most rows; the model can hit ~0.48 on them without semantic reasoning. The signal of real learning is in the new fields:
—outcome_of_latest_attempt: MRR 0.763 (2.76× random)
—recent_error: MRR 0.743 (2.90× random)
—initiating_command: MRR 0.667 (2.42× random)
—Not a standalone extractor. One head in a larger ingestion pipeline.

Related work

See docs/ranker-hypothesis-log-2026-05-08.md in the episodic-ingestion-compiler repo for the full experimental ladder.