nguha/legal_hallucinations_subset

Legal Hallucinations Subset

Dataset Description

This is a curated subset of the reglab/legal_hallucinations dataset, containing up to 1000 randomly sampled rows for each of 6 specific legal reasoning tasks (5444 rows total).

The original dataset was created for the paper: Dahl et al., "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models," Journal of Legal Analysis (2024, forthcoming). Preprint: arxiv:2401.01301

Dataset Details

Dataset Summary

This subset focuses on 6 specific legal reasoning tasks, with up to 1000 examples per task (5444 rows total). Each task is provided as a separate dataset split for easy access and evaluation.

Supported Tasks and Usage

The dataset contains the following splits (one per task):

• —affirm_reverse - Determining whether a court affirmed or reversed a lower court's decision
• —citation_retrieval - Retrieving correct legal citations
• —cited_precedent - Identifying cited legal precedents
• —court_id - Identifying the court that decided a case
• —majority_author - Identifying the author of a majority opinion
• —year_overruled - Identifying when a case was overruled

Dataset Structure

Each split contains the following columns:

• —task (string): The name of the task
• —query (string): The exact query/question submitted
• —example_correct_answer (string): An example of a correct answer to the query

Data Splits

Dataset Creation

Curation Process

1. 1.Source Data: Loaded from original_dataset.csv (subset of reglab/legal_hallucinations)
2. 2.Column Selection: Kept only task, query, and example_correct_answer columns
3. 3.Task Filtering: Filtered to only include the 6 specified tasks
4. 4.Quality Filtering: Removed rows with missing or empty example_correct_answer values
5. 5.Deduplication: Removed duplicate rows
6. 6.Sampling: Randomly sampled up to 1000 rows per task (using random seed 42 for reproducibility). Note: year_overruled has 444 examples as that's all that was available after filtering.

Filtering Criteria

• —Columns: Only task, query, and example_correct_answer are included
• —Tasks: Only the following 6 tasks are included:
• —affirm_reverse
• —citation_retrieval
• —cited_precedent
• —court_id
• —majority_author
• —year_overruled
• —Quality: All rows have non-empty example_correct_answer values
• —Deduplication: Duplicate rows have been removed
• —Sampling: Up to 1000 rows per task (or all available rows if fewer than 1000). year_overruled has 444 examples.

Usage

Loading the Dataset

from datasets import load_from_disk

# Load the entire dataset
dataset = load_from_disk("legal_hallucinations_subset")

# Access a specific task split
affirm_reverse_data = dataset["affirm_reverse"]
citation_retrieval_data = dataset["citation_retrieval"]

# Iterate over examples in a split
for example in affirm_reverse_data:
    print(f"Query: {example['query']}")
    print(f"Correct Answer: {example['example_correct_answer']}")

Example

from datasets import load_from_disk

dataset = load_from_disk("legal_hallucinations_subset")

# Get an example from the affirm_reverse split
example = dataset["affirm_reverse"][0]
print(example)
# {
#     'task': 'affirm_reverse',
#     'query': 'Did the court in ... affirm or reverse...?',
#     'example_correct_answer': 'affirm'
# }

Dataset Statistics

• —Total Rows: 5444 (1000 per task for 5 tasks, 444 for year_overruled)
• —Columns: 3 (task, query, examplecorrectanswer)
• —Splits: 6 (one per task)
• —Random Seed: 42 (for reproducibility)

Source and Citation

Source Dataset

This dataset is a subset of:

• —Dataset: reglab/legal_hallucinations
• —Repository: Stanford Regulation, Evaluation, and Governance Lab

Citation

If you use this dataset, please cite the original paper:

@article{dahl2024large,
  title={Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models},
  author={Dahl, Matt and Magesh, Varun and Suzgin, Mirac and Ho, Daniel E.},
  journal={Journal of Legal Analysis},
  year={2024},
  note={Forthcoming},
  arxiv={2401.01301}
}

License

[More Information Needed] - Please refer to the original dataset license.

Dataset Card Contact

For questions or issues related to this subset, please refer to the original dataset repository or open an issue.