Modelpublic

AmirMohseni/modernbert-large-v3-legal-guidance-prefix-len4096-seed42

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes24downloads

Model Card

ModernBERT-large for prefix-aware legal-guidance detection

This model detects whether legal guidance is already present in the cumulative user-turn prefix of a multi-turn LLM conversation. It was trained for research on early detection and routing of legal information needs. It is not a legal advice system and must not be used to determine whether a person has a valid legal claim.

Task formulation

One example is generated after each user turn. Prefixes before first_guidance_user_turn_id are negative; the onset prefix and all later prefixes are positive. Every prefix from a guidance-negative conversation is negative. The final prefix matches the user-only full-conversation serialization used in the comparison model.

All prefixes are retained. Each prefix is weighted by the inverse of its conversation's number of user turns, so every conversation has total training weight 1.0.

Data

—Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
—Requested revision: main (latest at run time)
—Resolved revision: 489b87c18e571e8c7e9a3b1c2463b79e50cef120
—Training: 1,632 conversations / 4,865 prefixes
—Validation: 290 conversations / 912 prefixes
—Positive prefixes: 2,042 train / 376 validation

The first-guidance-turn annotations are silver labels and were not independently human-validated for this run. Onset metrics must therefore not be described as gold evaluation.

Training configuration

Setting	Value
Base model	`answerdotai/ModernBERT-large`
Resolved base revision	`45bb4654a4d5aaff24dd11d4781fa46d39bf8c13`
GPU	`NVIDIA A100-SXM4-40GB` (39.5 GiB)
Maximum length	4096
Epochs	3
Learning rate	5e-05
Effective batch size	32
Weight decay	0.01
Seed	42
Loss weighting	Equal total weight per conversation
W&B run	https://wandb.ai/rl-research-team/legal-guidance-prefix-v3/runs/u0m7x6qb

Validation results

The threshold 0.36 was selected on the silver validation split by conversation-weighted prefix macro-F1, with positive-class F1 as tie-breaker.

Evaluation	Macro-F1	Positive F1	AUPRC	Accuracy	Balanced accuracy
All prefixes, conversation-weighted	0.8741	0.8529	0.9181	0.8777	0.8782
All prefixes, unweighted	0.8385	0.8144	0.8973	0.8421	0.8419
Final/full prefixes	0.8753	0.8667	0.9344	0.8759	0.8757

Silver onset analysis

Measure	Value
Negative-conversation false-alarm rate	0.1410
Pre-onset false-alarm rate	0.1119
Missed-guidance rate	0.1045
Exact onset accuracy	0.7388
Within-one-user-turn accuracy	0.8060
Mean absolute turn error among detected positives	0.3500

Repository artifacts

—prefix_validation_metrics.json: complete metrics and threshold curve
—prefix_validation_predictions.csv: per-prefix probabilities and predictions
—onset_validation_predictions.csv: conversation-level onset predictions
—run_metadata.json: model, data, software, and hardware provenance

Limitations

The dataset is English-language, jurisdiction-agnostic, and derived from public LLM interaction logs that may contain sensitive content. The training and onset labels are silver labels. The model was trained with one seed and has not yet been evaluated on the adjudicated 200-conversation human gold set. Final-prefix guidance evaluation on gold is appropriate after adjudication; onset evaluation on gold requires separate human review of first_guidance_user_turn_id.