jurisbenchomni-anonymous/jurisbenchomni

JurisBenchOmni — Model Predictions

This repository hosts the per-sample model predictions and the analysis-ready aggregated CSVs that accompany our paper JurisBenchOmni: A Judicial-Practice-Oriented Benchmark for Legal Omni-Modal Evaluation.

What you get here lets you read off, slice, and re-aggregate the per-task / per-dimension / pipeline-level numbers we cite in the paper, without having to re-run inference. Companion code that produces the aggregated tables from the per-sample JSON files lives in our GitHub repository.

• —This dataset: <https://huggingface.co/datasets/jurisbenchomni-anonymous/jurisbenchomni>
• —Companion code: <https://github.com/jurisbenchomni/jurisbenchomni>

Layout

.
├── README.md
├── predictions/         3,300 raw model outputs, one file per (model, task, case)
│   ├── G3P/             gemini-3.1-pro-preview, all 11 tasks
│   │   ├── R1/<case_id>.json         the prediction
│   │   ├── R1/<case_id>.prompt.json  the exact prompt that produced it
│   │   ├── R2/...  S1/...  S2/...  S3/...
│   │   ├── I1/...  I2/...  I3/...
│   │   └── G1/...  G2/...  G3/...
│   └── Q35O/            qwen-3.5-omni-flash, all 11 tasks
├── metadata/
│   ├── cases.csv        the 150 hearing cases (case_index, case_id)
│   ├── tasks.csv        the 11 tasks (paper Table 1) with dim & granularity
│   └── coverage.csv     which (model × task) cells have data
└── aggregated/          the 10 analyzed CSVs (long format + summary tables)
    ├── all_samples.csv             1 row per prediction (source of truth)
    ├── by_task_model.csv           wide pivot: per (task × model) accuracy
    ├── by_dim_model.csv            wide pivot: per (dim × model) accuracy
    ├── case_pass.csv               per (case, model): dim-level pass flag
    ├── coverage.csv                (model × task) sample count + parser errors
    ├── task_meta.csv               static paper Table 1 metadata
    ├── summary_with_ci.csv         per (model, task) accuracy + 95% Wilson CI + Random / Majority baselines
    ├── dimension_summary.csv       per (model, dimension) accuracy + CI
    ├── granularity_summary.csv     per (model, granularity) accuracy + CI
    └── baselines.csv               per task: random_acc, majority_acc, majority_letter

Conventions

• —Models. G3P = gemini-3.1-pro-preview, Q35O = qwen-3.5-omni-flash.
• —Tasks. R1, R2, S1, S2, S3, I1, I2, I3, G1, G2, G3 — see metadata/tasks.csv and the paper §3 / Table 1 for definitions.
• —Modality. All predictions are from the V configuration (video with embedded audio).
• —Case IDs are written in their original Chinese full-width form (e.g. （2018）云0111民初9891号). UTF-8 throughout.
• —Accuracy. correct = exact_set_match; parser-failed predictions count as wrong. A recovery fallback salvages per-option JSON dicts ({"A":0,"B":1,...}) that the upstream Qwen evaluator could not reduce to a single letter; rows where this happened have recovered_from_text = true in aggregated/all_samples.csv and the original error string is preserved in parse_error_orig.
• —Total sample count. 3,300 = 150 cases × 11 tasks × 2 models. Both models cover the full benchmark.

Field schema for predictions/<model>/<task>/<case>.json

Each prediction file is the original per-sample JSON with two opaque fields stripped to keep the release small and reviewable:

• —raw_response.sdk_http_response — removed (HTTP headers, server timing).
• —raw_response.candidates[*].content.parts[*].thought_signature — removed (Gemini-internal opaque base64 token).

All evidence is preserved: prediction, gold, exact_set_match, parse_error, the model's free-form text, finish_reason, usage, model_version, response_id, latency_ms, prompt_meta.

Loading

For programmatic access via datasets:

from datasets import load_dataset
# main long-format predictions table:
ds = load_dataset("jurisbenchomni-anonymous/jurisbenchomni",
                   data_files="aggregated/all_samples.csv", split="train")

For random-access reading of an individual prediction with its full raw response, read the JSON file directly under predictions/<model>/<task>/.

Privacy / data release

For privacy reasons, the source court-hearing videos and the corresponding written judgment documents are not included in this release. What this repository contains is limited to:

• —the per-sample model predictions and prompts,
• —the per-task / per-dimension aggregated CSVs,
• —the case-level metadata required to interpret them.

The hearing videos and judgment documents involve identifiable participants (judges, plaintiffs, defendants, counsel) and case-specific factual records. Distributing them under the standard public-dataset model would be inconsistent with our anonymity and de-identification commitments.

After the paper is accepted, we plan to release the underlying hearing videos and judgment documents through a controlled-access protocol that satisfies the relevant privacy and ethics requirements and that guarantees the long-term availability of the benchmark. We will update this dataset card with the access procedure at that point.

Because every prediction in this release was generated by feeding a real court-hearing clip (with embedded audio) and the corresponding case metadata to the evaluated model, the numbers cited in the paper are fully reproducible from this release once the access protocol for the videos and judgments is in place.