NoeFlandre/fineweb-legal-pilot

⚖️ FineWeb-Legal-Pilot

<center> <img src="https://raw.githubusercontent.com/NoeFlandre/fineweb-legal/main/assets/logo.png" alt="FineWeb-Legal: The finest collection of legal content the web has to offer" width="400"/> </center>

66.8M words of the finest legal domain data the 🌐 web has to offer.

Repo: GitHub | Report: Technical Report

What is it?

FineWeb-Legal-Pilot is a pilot dataset consiting of 52k high-quality legal documents filtered from the 10-billion-token sample of 🍷 FineWeb.

To enhance FineWeb's utility for legal AI domain adaptation, we draw inspiration from the FineWeb-Edu methodology: creating a legal quality classifier using annotations generated by Mistral-Medium. We then used this classifier to score and retain only the most legally significant web pages (case law, statutes, filings).

This pilot release serves as a validation step before scaling the pipeline to the full 44TB FineWeb corpus.

What is being released?

Along with the dataset, we are releasing the legal classifier model used for filtering, as well as the training code and annotation pipeline:

• —Dataset (Filtered): This repo (fineweb-legal-pilot)
• —Dataset (Annotations): NoeFlandre/fineweb-legal-annotations
• —Classifier: HuggingFaceFW/fineweb-legal-classifier
• —Code: GitHub Repository

How to load the dataset

You can load the full dataset (default) or one of the specialized high-quality subsets.

Configurations

• —default: All identified legal documents. Broadest coverage.
• —high_quality: Strong legal content. Case law, statutes, contracts.
• —supreme: Gold Standard. Supreme Court opinions, primary legislation.

Using datasets

from datasets import load_dataset

# Load the "Supreme" Gold Standard configuration
ds = load_dataset("NoeFlandre/fineweb-legal-pilot", "supreme", split="train")

# Load the default configuration
ds_default = load_dataset("NoeFlandre/fineweb-legal-pilot", "default")

Dataset Curation

We adopted the synthetic data annotation approach pioneered by FineWeb-Edu.

Annotation

We used Mistral-Medium to score 6,500 FineWeb samples on a scale of 0 to 5, based on their legal value.

Classifier Training

We fine-tuned a Gemma-Embedding-300m model using LoRA adapters on these annotations. The model achieves:

• —Binary F1@3: 97.99%
• —Validation Accuracy: 88.8%

The training process took ~2 hours on a single RTX 3090.

Filtering Results

We filtered the FineWeb 10BT sample (approx 143k candidates pre-filtered by heuristics) using this classifier.

• —Input: 143,379 documents (extracted by heuristics from the original FineWeb 10BT subset)
• —Output: 52,132 documents (Score ≥ 3.0)
• —Top domains include openjurist.org, findacase.com, and federalregister.gov.

Considerations for Using the Data

Social Impact

Access to high-quality legal training data is currently restricted to proprietary databases (Westlaw, LexisNexis). By releasing FineWeb-Legal, we aim to democratize access to legal AI research, enabling the community to train open legal LLMs.

Biases

The dataset reflects the bias of the web and Common Crawl. While heavily filtered for legal content, it may still contain historical legal documents with outdated or offensive terminology typical of their era.

Limitations

This is a Pilot release derived from only 10 billion tokens. It is not yet large enough for full pre-training but is ideal for:

• —Domain adaptation (fine-tuning)
• —Evaluation benchmarks
• —Training retrieval (RAG) embeddings

Additional Information

Licensing

The dataset is released under the MIT License and adheres to the Open Data Commons Attribution License (ODC-By) v1.0 of the parent FineWeb dataset.

Citation

@misc{fineweb-legal-2026,
    author       = { Noé Flandre },
    title        = { FineWeb-Legal-Pilot: High-Quality Legal Text from the Web },
    year         = 2026,
    url          = { https://github.com/NoeFlandre/fineweb-legal },
    publisher    = { Hugging Face }
}