CoolFace
Modelpublic

Nahla-yasmine/legal-bert-peft-eurlex

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
0likes5downloads
Model Card

Legal-BERT-PEFT-EURLEX

A BERT model fine-tuned on EU legal documents from the Pile of Law dataset using Parameter-Efficient Fine-Tuning (PEFT).

Model Details

Base Model

  • —Architecture: bert-base-uncased
  • —Parameters: 110 million
  • —Language: English

Fine-tuning Details

  • —Method: PEFT with LoRA (Low-Rank Adaptation)
  • —Trainable Parameters: 1.3 million (1.21% of total)
  • —Training Approach: Masked Language Modeling (MLM)

Training Data

  • —Dataset: EURLEX subset of Pile of Law
  • —Training Samples: 20,000 legal documents
  • —Domain: European Union Legal Documents
  • —Text Length: Average 13,327 characters per document

Performance

Training Results

MetricBase ModelFine-tuned ModelImprovement
Test Loss1.93270.658066.69%
Perplexity6.911.9372.07%

Training Configuration

Hyperparameters

ParameterValue
Learning Rate2e-4
Batch Size16 (8 × gradient accumulation 2)
Epochs3
Max Sequence Length512
Warmup Steps500
Weight Decay0.01

Intended Use Cases

Recommended Use

  • —Legal document analysis and processing
  • —Masked language modeling in legal contexts
  • —Legal text understanding and generation
  • —Research in computational law and legal AI
  • —Educational purposes in legal technology

Limitations and Bias

Limitations

  • —Domain Specific: Primarily effective on legal text, especially EU law
  • —Language: English only
  • —Scope: Trained on a subset of EURLEX documents
  • —Temporal Scope: Training data up to 2022 only

Qualitative Examples

Example 1: Legal Judgment

text
Input: "The court found the defendant [MASK] of all charges."
Predictions: ["guilty", "innocent", "acquitted", "free", "liable"]

Example 2: Contract Law

text
Input: "The contract was declared [MASK] due to fraudulent activities."
Predictions: ["void", "invalid", "null", "bankrupt", "cancelled"]

Example 3: Civil Law

text
Input: "The plaintiff sought [MASK] for damages incurred."
Predictions: ["compensation", "damages", "only", "insurance", "forgiveness"]

Usage

python
from transformers import AutoModelForMaskedLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForMaskedLM.from_pretrained("Nahla-yasmine/legal-bert-peft-eurlex")
tokenizer = AutoTokenizer.from_pretrained("Nahla-yasmine/legal-bert-peft-eurlex")

# Example: Masked language prediction
text = "The court found the defendant [MASK] of all charges."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Get top predictions
mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
logits = outputs.logits[0, mask_token_index, :]
top_tokens = torch.topk(logits, 5, dim=1).indices[0].tolist()

for i, token_id in enumerate(top_tokens):
    predicted_token = tokenizer.decode([token_id])
    print(f"{i+1}. {predicted_token}")

Advanced Usage with PEFT

python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForMaskedLM

# Load base model
base_model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

# Load PEFT adapter
model = PeftModel.from_pretrained(base_model, "Nahla-yasmine/legal-bert-peft-eurlex")

@software{legalbertpeft_2024, title = {Legal-BERT-PEFT-EURLEX}, author = {Nahla-yasmine}, year = {2024}, url = {https://huggingface.co/Nahla-yasmine/legal-bert-peft-eurlex} }

  • —PEFT 0.17.0