Modelpublic

Avifenesh/episodic-ingestion-modernbert-field-event-ranker-mixed-v2-1-h4-320

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes10downloads

Model Card

ModernBERT field-event ranker (mixed-mode v2.1, H4-320)

Fine-tune of answerdotai/ModernBERT-base with v2.1 labels — same as v2 but with a sharpened `action_causality` heuristic:

—Old: tool_result is grounded if ≥2 content tokens appear in ANY later assistant turn. Noise-dominated (file, path, error, exitcode match trivially).
—New: require ≥2 distinctive tokens (filter out 50+ corpus-frequent noise tokens) and the assistant turn must be the NEXT one, within 2 events. Selects a proper subset of tool_results based on content grounding.

Key result: per-field improvements on sharpened fields

Field	v2-320	v2.1-320	Δ
action_causality	0.561	0.698	+0.137
failed_attempts	0.705	0.845	+0.140
recent_error	0.826	0.905	+0.079
customer_identity	0.811	0.864	+0.053
next_actions	0.755	0.786	+0.031
initiating_command	0.966	0.968	+0.002
outcomeoflatest_attempt	0.966	0.938	-0.028
attempted_actions	0.480	0.473	-0.007
observed_outcomes	0.491	0.486	-0.005

Overall MRR stayed flat at 0.583 (aggregate), but top-1 moved from 0.375 → 0.381. action_causality's eval mass shrank (141 → 50 pairs) because the sharper heuristic only emits the field when the signal is real, which is why the big per-field MRR gain doesn't translate 1:1 into aggregate gain.

Training details

—Train rows: 1217
—Eval rows: 220
—Steps: 320 × accum=8 = 2560 forwards
—Last loss: 0.0007
—Peak VRAM: 4.08 GiB

Eval metrics

metric	value
overall MRR	0.583
top-1	0.381
top-2	0.576
top-3	0.722
top-5	0.904
mean expected rank	2.69

Per-field MRR (all fields):

field	n	MRR	top-1
`attempted_actions`	846	0.473	0.24
`observed_outcomes`	845	0.486	0.24
`initiating_command`	199	0.968	0.94
`outcome_of_latest_attempt`	181	0.938	0.88
`attempt_outcome_pairs`	53	0.608	0.34
`action_causality`	50	0.698	0.48
`failed_attempts`	36	0.845	0.72
`next_actions`	32	0.786	0.59
`recent_error`	29	0.905	0.83
`customer_identity`	11	0.864	0.73
`transaction_reference`	5	0.583	0.40
`product_name`	4	1.000	1.00
`discarded_options`	2	0.292	0.00
`invalidation_hints`	2	0.625	0.50
`non_promotable_context`	2	0.375	0.00
`payment_or_warranty_detail`	2	0.170	0.00
`assistant_claims_to_verify`	1	1.000	1.00
`explicit_decisions`	1	1.000	1.00
`resolved_context`	1	0.333	0.00
`touched_files`	1	0.333	0.00
`unsupported_hypotheses`	1	0.250	0.00

Lineage

checkpoint	MRR	top-1	note
mixed-v1 perf-H4	0.506	0.260	role-tautological labels
mixed-v2 perf-H4 (160)	0.526	0.296	semantic labels, non-converged
mixed-v2 H4-320	0.583	0.375	converged
mixed-v2.1 H4-320 (this)	0.583	0.381	sharpened causality