Whisler/camembert_finetuned_progressive

CamemBERT Finetuned Progressive

CamemBERT finetuned on 22,000 French parliamentary Q&A pairs for improved embedding in Retrieval-Augmented Generation (RAG) systems.

🤖 Model Type: Feature extraction & embedding 🇫🇷 Language: French 📜 Training Data: 22,000 Q&A pairs from the French Parliament (2017-2025) ⚖️ Use Case: Legal/regulatory analysis, parliamentary response drafting, semantic search in administrative documents

Model Details

Description

This model is a fine-tuned version of `camembert-base` optimized for embedding French parliamentary questions and answers. It is designed to improve the performance of RAG systems in legal and administrative contexts, such as:

• —Automated drafting of parliamentary responses.
• —Retrieval of relevant legal articles or budgetary information.
• —Semantic search in administrative documents.

Training Data

The model was fine-tuned on a dataset of 22,000 Q&A pairs from the French Parliament (Assemblée Nationale and Sénat). The dataset includes questions from MPs and responses from the Government, covering topics such as:

• —Legal and regulatory issues.
• —Budgetary and financial matters.
• —Public policies and administrative procedures.

Training Procedure

• —Base Model: camembert-base
• —Fine-tuning Task: Masked Language Modeling (MLM)
• —Context Preservation: Special attention was given to maintaining the legal and administrative context (e.g., references to laws, decrees, and ministries).
• —Chunking: Texts were split into chunks with overlapping context to preserve semantic coherence.

Uses

Direct Use

This model generates 768-dimensional embeddings that can be used in:

• —RAG pipelines (e.g., with Qdrant, Weaviate, or Elasticsearch).
• —Semantic search in legal or administrative documents.
• —Automated drafting tools for parliamentary responses.

Downstream Use

• —Public Administration: Assisting civil servants in drafting responses to parliamentary questions.
• —Legal Tech: Enhancing search and analysis in legal databases.
• —Research: Studying French legislative language and discourse.

Out-of-Scope Use

• —Non-French Texts: The model is specialized for French and may underperform on other languages.
• —General-Purpose Embeddings: While it can be used for general French text, it is optimized for legal/administrative language.

Performance

• —15-20% improvement in retrieval relevance compared to camembert-base on a validation set of 2,000 parliamentary questions.
• —Robustness: Handles complex formulations, such as references to multiple laws or budgetary lines.

How to Use

Installation

pip install transformers torch