Datasetpublic

ai-law-society-lab/oral-args-data-and-results

Oral Arguments Arena Data repository for AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments (Zhang, Nadeem, Zheng, Stammbach, Henderson, 2026). Refer to the paper for background on the evaluation framework, experimental design, and findings. Repository Structure oral-args-arena-annotations/ ├── transcript_data/ # SCOTUS oral argument transcripts and case briefs ├── automated_metrics/ # LLM classifier outputs… See the full description on the dataset page: https://huggingface.co/datasets/ai-law-society-lab/oral-args-data-and-results.

sourceHugging Facecc-by-4.0updated 6mo agoView on Hugging Face

1likes95downloads

Dataset Card

Oral Arguments Arena

Data repository for **AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments** (Zhang, Nadeem, Zheng, Stammbach, Henderson, 2026). Refer to the paper for background on the evaluation framework, experimental design, and findings.

Repository Structure

oral-args-arena-annotations/
├── transcript_data/           # SCOTUS oral argument transcripts and case briefs
├── automated_metrics/         # LLM classifier outputs (SQLite databases)
└── human_annotations/         # Human evaluation data and analysis

Each subdirectory contains its own README with detailed documentation.

`transcript_data/`

2,488 cleaned SCOTUS oral argument transcripts (1991–2025) sourced from the Oyez API, along with corresponding case briefs. Includes notebooks for transcript cleaning and SQLite database creation. See `transcript_data/README.md`.

`automated_metrics/`

SQLite databases containing LLM classifier annotations of generated judicial remarks across distributional metrics, adversarial/decorum metrics, issue coverage, and logical fallacy detection. All .db files are tracked via Git LFS.

`human_annotations/`

Human evaluation data from 7 annotators across three annotation tasks: distributional metric classification (T1), side-by-side model comparisons (T2), and arena-style pairwise preference voting. Includes inter-rater agreement analysis. See `human_annotations/README.md`.

Setup

All .db files are tracked with Git LFS.

bash

git lfs install

Citation

bibtex

@article{zhang2026aiassisted,
  title={AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments},
  author={Zhang, Kylie and Nadeem, Nimra and Zheng, Lucia and Stammbach, Dominik and Henderson, Peter},
  journal={arXiv preprint arXiv:2603.04718},
  year={2026}
}