Abbasgamer1/legalMind

Model Card: final-merged-model3-pruned

Introduction

This model card describes the parameters, training, and evaluation of the final-merged-model3-pruned model, a modified BERT architecture for sequence classification tasks. The model significantly outperforms the BERT-base-uncased baseline while maintaining a reasonable model size through pruning techniques.

Model Details

Layer Distribution

Training Information

Training Process

• —Training Framework: PyTorch
• —Optimization Algorithm: AdamW
• —Learning Rate Schedule: Linear warmup and decay
• —Batch Size: 32
• —Hardware: NVIDIA A100 GPUs
• —Training Time: Approximately 12 hours

Training Metrics

Pruning Process

The model underwent a layer-based pruning process to reduce its size while maintaining performance:

1. 1.Original model size: 6.60 GB
2. 2.Pruned model size: 4.71 GB
3. 3.Size reduction: 28.6%

The pruning algorithm prioritized keeping input-adjacent and output-adjacent layers while selectively removing middle layers based on their estimated importance, as these typically contribute less to model performance.

GLUE Benchmark Performance

Inference Performance

• —Recommended Hardware: NVIDIA V100 or newer
• —Minimum RAM: 16GB
• —Average Inference Time: 45ms per sequence
• —Throughput: ~22 sequences per second

Limitations and Biases

• —The model inherits biases present in its base BERT architecture
• —Limited evaluation on non-English texts
• —Increased computational requirements compared to smaller models
• —Not optimized for edge devices due to size

Intended Use

• —High-accuracy sequence classification tasks
• —Legal document analysis
• —Academic text processing
• —Applications where accuracy is prioritized over inference speed

Comparison to BERT-base-uncased

Citations

@article{our_model2025,
  title={Improving BERT Performance through Selective Layer Pruning},
  author={Author, A. and Author, B.},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  year={2025},
  volume={},
  number={},
  pages={},
  publisher={IEEE}
}

@article{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1810.04805},
  year={2018}
}

Model Overview

Model Name: LegalMind Merged Model 3Model Type: Text ClassificationBase Model: BERT-base-uncasedNumber of Labels: 2Merged Models: Combination of multiple fine-tuned .h5 and .safetensors modelsFramework: PyTorch, Transformers (Hugging Face)

Model Description

This model is a fine-tuned BERT-based sequence classification model designed for legal document classification tasks. It has been trained on a mixture of datasets and optimized for real-world applications in the LegalMind project. The final model is an ensemble of multiple .h5 and .safetensors models, merged to leverage knowledge from multiple fine-tuned versions.

Training Details

Dataset: Fine-tuned on legal text classification datasets

Preprocessing: Tokenized using bert-base-uncased tokenizer

Loss Function: Cross-entropy loss

Optimizer: AdamW

Batch Size: 16

Learning Rate: 5e-5

Max Sequence Length: 128

Model Usage

How to Use

from transformers import AutoTokenizer, BertForSequenceClassification import torch

tokenizer = AutoTokenizer.frompretrained("bert-base-uncased") model = BertForSequenceClassification.frompretrained("pathtomodel")

def classifytext(text): inputs = tokenizer(text, returntensors="pt", truncation=True, padding=True, maxlength=128) with torch.nograd(): outputs = model(**inputs) logits = outputs.logits prediction = torch.argmax(logits, dim=-1).item() return prediction

text = "Example legal document text." print("Predicted Class:", classify_text(text))

Our Model 2 = This is trained with our datasets and has been merged with other best models bringing our Accuracy to almost 98% Our Model 3 = This is our trained model 2 merged with Deepseek R1 - 7B

Inference API

If hosted on Hugging Face:

import requests APIURL = "https://api-inference.huggingface.co/models/Abbasgamer1/legalMind" headers = {"Authorization": "Bearer YOURHF_TOKEN"}

def query(text): payload = {"inputs": text} response = requests.post(API_URL, headers=headers, json=payload) return response.json()

print(query("Example legal document text."))

Model Limitations

Requires GPU for fast inference.

Performance depends on fine-tuning quality and data.

May not generalize well to non-legal text.