ddrg/super_eurlex

Dataset Card for SuperEURLEX

This dataset contains over 4.6M Legal Documents from EURLEX with Annotations. Over 3.7M of this 4.6M documents are also available in HTML format. This dataset can be used for pretraining language models as well as for testing them on legal text classification tasks.

Use this dataset as follows:

from datasets import load_dataset
config = "0.DE" # {sector}.{lang}[.html]
dataset = load_dataset("ddrg/super_eurlex", config, split='train')

Dataset Details

Dataset Description

This Dataset was scrapped from EURLEX. It contains more than 4.6M Legal Documents in Plain Text and over 3.7M In HTML Format. Those Documents are separated by their language (This Dataset includes a total of 24 official European Languages) and by their Sector.

The Table below shows the number of documents per language:

• —Curated by: [More Information Needed]
• —Funded by [optional]: [More Information Needed]
• —Shared by [optional]: [More Information Needed]
• —Language(s) (NLP): [More Information Needed]
• —License: [More Information Needed]

Dataset Sources [optional]

• —Repository: https://huggingface.co/datasets/ddrg/super_eurlex/tree/main
• —Paper [optional]: [More Information Needed]
• —Demo [optional]: [More Information Needed]

Uses

As Corpus for:

• —Pretraining of Language Models with self supervised tasks like Masked Language Modeling and Next Sentence Prediction
• —Legal Text Analysis

As Dataset for evaluation on the following task:

• —eurovoc-Concepts Prediction i.e. which tags apply? (Muli-Label Classification (large Scale))
• —Example for this task is given[below
• —subject-matter Prediction i.e. which other tags apply (Multi-Label Classification)
• —form Classification i.e. What Kind of Document is it? (Multi-Class)
• —And more

Example for Use Of EUROVOC-Concepts

from datasets import load_dataset
import transformers as tr
from sklearn.preprocessing import MultiLabelBinarizer
import numpy as np 
import evaluate
import uuid

# ==================== #
#     Prepare Data     #
# ==================== #
CONFIG = "3.EN" # {sector}.{lang}[.html]
MODEL_NAME = "distilroberta-base"
dataset = load_dataset("ddrg/super_eurlex", CONFIG, split='train')
tokenizer = tr.AutoTokenizer.from_pretrained(MODEL_NAME)

# Remove Unlabeled Columns
def remove_nulls(batch):
  return [(sample != None) for sample in batch["eurovoc"]]
dataset = dataset.filter(remove_nulls, batched=True, keep_in_memory=True)

# Tokenize Text
def tokenize(batch):
  return tokenizer(batch["text_cleaned"], truncation=True, padding="max_length")
# Keep in Memory is optional (The Dataset is large though and can easily use up alot of memory)
dataset = dataset.map(tokenize, batched=True, keep_in_memory=True)

# Create Label Column by encoding Eurovoc Concepts 
encoder = MultiLabelBinarizer()
# List of all Possible Labels 
eurovoc_concepts = dataset["eurovoc"]
encoder.fit(eurovoc_concepts)
def encode_labels(batch):
    batch["label"] = encoder.transform(batch["eurovoc"])
    return batch
dataset = dataset.map(encode_labels, batched=True, keep_in_memory=True)

# Split into train and Test set
dataset = dataset.train_test_split(0.2)

# ==================== #
#  Load & Train Model  #
# ==================== #
model = tr.AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME,
    num_labels=len(encoder.classes_),
    problem_type="multi_label_classification",
)

metric = evaluate.load("JP-SystemsX/nDCG", experiment_id=uuid.uuid4())
def compute_metric(eval_pred):
    predictions, labels = eval_pred
    return metric.compute(predictions=predictions, references=labels, k=5)

# Set Hyperparameter 
# Note: We stay mostly with default values to keep example short
# Though more hyperparameter should be set and tuned in praxis
train_args = tr.TrainingArguments(
    output_dir="./cache",
    per_device_train_batch_size=16,
    num_train_epochs=20
)
trainer = tr.Trainer(
    model=model,
    args=train_args,
    train_dataset=dataset["train"],
    compute_metrics=compute_metric,
)
trainer.train() # This will take a while
print(trainer.evaluate(dataset["test"]))
# >>> {'eval_loss': 0.0018887673504650593, 'eval_nDCG@5': 0.8072531683578489, 'eval_runtime': 663.8582, 'eval_samples_per_second': 32.373, 'eval_steps_per_second': 4.048, 'epoch': 20.0}

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

[More Information Needed]

Dataset Structure

This dataset is divided into multiple split by Sector x Language x Format

Sector refers to the kind of Document it belongs to:

• —0: Consolidated acts
• —1: Treaties
• —2: International agreements
• —3: Legislation
• —4: Complementary legislation
• —5: Preparatory acts and working documents
• —6: Case-law
• —7: National transposition measures
• —8: References to national case-law concerning EU law
• —9: Parliamentary questions
• —C: Other documents published in the Official Journal C series
• —E: EFTA documents

Language refers to each of the 24 official European Languages that were included at the date of the dataset creation:

• —BG ~ Bulgarian
• —CS ~ Czech
• —DA ~ Danish
• —DE ~ German
• —EL ~ Greek
• —EN ~ English
• —ES ~ Spanish
• —ET ~ Estonian
• —FI ~ Finnish
• —FR ~ French
• —GA ~ Irish
• —HR ~ Croatian
• —HU ~ Hungarian
• —IT ~ Italian
• —LT ~ Lithuanian
• —LV ~ Latvian
• —MT ~ Maltese
• —NL ~ Dutch
• —PL ~ Polish
• —PT ~ Portuguese
• —RO ~ Romanian
• —SK ~ Slovak
• —SL ~ Slovenian
• —SV ~ Swedish

Format refers to plain Text (default) or HTML format (.html)

Note: Plain Text contains generally more documents because not all documents were available in HTML format but those that were are included in both formats

Those Splits are named the following way: {sector}.{lang}[.html]

For Example:

• —3.EN would be English legislative documents in plain text format
• —3.EN.html would be the same in HTML Format

Each Sector has its own set of meta data:

<details><summary>Sector 0 (Consolidated acts)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty

</p> </details>

<details><summary>Sector 1 (Treaties)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —currentconsolidatedversion ~ date when this version of the document was consolidated Format DD/MM/YYYY
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information

</p> </details>

<details><summary>Sector 2 (International agreements)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information
• —latestconsolidatedversion ~ Format DD/MM/YYYY
• —currentconsolidatedversion ~ Format DD/MM/YYYY

</p> </details>

<details><summary>Sector 3 (Legislation)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information
• —latestconsolidatedversion ~ Format DD/MM/YYYY
• —currentconsolidatedversion ~ Format DD/MM/YYYY

</p> </details>

<details><summary>Sector 4 (Complementary legislation)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information
• —latestconsolidatedversion ~ Format DD/MM/YYYY
• —currentconsolidatedversion ~ Format DD/MM/YYYY

</p> </details>

<details><summary>Sector 5 (Preparatory acts and working documents)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information
• —latestconsolidatedversion ~ Format DD/MM/YYYY

</p> </details>

<details><summary>Sector 6 (Case-law)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information
• —case-lawdirectorycodebeforelisbon ~ Classification system used for case law before Treaty of Lisbon came into effect (2009), each code reflects a particular area of EU law

</p> </details>

<details><summary>Sector 7 (National transposition measures)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —transposedlegalacts ~ national laws that exist in EU member states as a direct result of the need to comply with EU directives

</p> </details>

<details><summary>Sector 8 (References to national case-law concerning EU law)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —case-lawdirectorycodebeforelisbon ~ Classification system used for case law before Treaty of Lisbon came into effect (2009), each code reflects a particular area of EU law
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information

</p> </details>

<details><summary>Sector 9 (Parliamentary questions)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information

</p> </details>

<details><summary>Sector C (Other documents published in the Official Journal C series)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information

</p> </details>

<details><summary>Sector E (EFTA documents)</summary><p>

• —celexid_ ~ Unique Identifier for each document
• —textcleaned (Plain Text) **or** texthtmlraw_ (HTML Format)
• —form ~ Kind of Document e.g. Consolidated text, or Treaty
• —directorycode_ ~ Information to structure documents in some kind of directory structure by topic e.g. '03.50.30.00 Agriculture / Approximation of laws and health measures / Animal health and zootechnics'
• —subjectmatter ~ Keywords that provide general overview of content in a document see [here](https://eur-lex.europa.eu/content/e-learning/browsingoptions.html) for more information
• —eurovoc ~ Keywords that describe document content based on the European Vocabulary see here for more information

</p> </details>

Dataset Creation

Curation Rationale

This dataset was created for the creation and/or evaluation of pretrained Legal Language Models.

Source Data

Data Collection and Processing

We used the EURLEX-Web-Scrapper Repo for the data collection process.

Who are the source data producers?

The Source data stems from the EURLEX-Website and was therefore produced by various entities within the European Union

Personal and Sensitive Information

No Personal or Sensitive Information is included to the best of our knowledge.

Bias, Risks, and Limitations

• —We removed HTML documents from which we couldn't extract plain text under the assumption that those are corrupted files. However, we can't guarantee that we removed all.
• —The Extraction of plain text from legal HTML documents can lead to formatting issues e.g. the extraction of text from tables might mix up the order such that it becomes nearly incomprehensible.
• —This dataset might contain many missing values in the meta-data columns as not every document was annotated in the same way

[More Information Needed]

Recommendations

• —Consider Removing rows with missing values for the task before training a model on it

Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Dataset Card Authors [optional]

[More Information Needed]

Dataset Card Contact

[More Information Needed]