AgentPublic/dole

1---2language:3- fr4tags:5- france6- legislation7- law8- embeddings9- open-data10- government11- parlement12pretty_name: French Legislative Dossiers Dataset (DOLE)13size_categories:14- 1K<n<10K15license: etalab-2.016configs:17- config_name: latest18  data_files: "data/dole-latest/*.parquet"19  default: true20---21---------------------------------------------------------------------------------------------------22### 📢 Sondage 2026 : Utilisation des datasets publiques de MediaTech23Vous utilisez ce dataset ou d’autres datasets de notre collection [MediaTech](https://huggingface.co/collections/AgentPublic/mediatech) ? Votre avis compte ! 24Aidez-nous à améliorer nos datasets publiques en répondant à ce sondage rapide (5 min) : 👉 https://grist.numerique.gouv.fr/o/albert/forms/gF4hLaq9VvUog6c5aVDuMw/11 25Merci pour votre contribution ! 🙌26 27---------------------------------------------------------------------------------------------------28# 🇫🇷 French Legislative Dossiers Dataset (DOLE)29 30This dataset provides a semantic-ready, chunked and embedded version of the **Dossiers Législatifs** ("DOLE") published by the French government. It includes all **laws promulgated since the XIIᵉ legislature (June 2002)**, **ordinances**, and **legislative proposals** under preparation.31The original data is downloaded from [the dedicated **DILA** open data repository](https://echanges.dila.gouv.fr/OPENDATA/DOLE) and is also published on [data.gouv.fr](https://www.data.gouv.fr/datasets/dole-les-dossiers-legislatifs/).32 33Each article is chunked and vectorized using the [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) embedding model, enabling use in **semantic search**, **retrieval-augmented generation (RAG)**, and **legal research** systems for example.34 35---36 37## 🗂️ Dataset Contents38 39The dataset is available in **Parquet format** and contains the following columns:40 41| Column Name         | Type             | Description                                                                 |42|---------------------|------------------|-----------------------------------------------------------------------------|43| `chunk_id`          | `str`            | Unique identifier for each chunk.                                           |44| `doc_id`            | `str`            | Document identifier from the source site.                                   |45| `chunk_index`       | `int`            | Index of the chunk within its original document. Starting from 1.           |46| `chunk_xxh64`       | `str`            | XXH64 hash of the `chunk_text` value.                                       |47| `category`          | `str`            | Type of dossier (e.g., `LOI_PUBLIEE`, `PROJET_LOI`, etc.).                  |48| `content_type`      | `str`            | Nature of the content: `article`, `dossier_content`, or `explanatory_memorandum`. |49| `title`             | `str`            | Title summarizing the subject matter.                                       |50| `number`            | `str`            | Internal document number.                                                   |51| `wording`           | `str`            | Libelle, Legislature reference (e.g., `XIVème législature`).                |52| `creation_date`     | `str`            | Creation or publication date (YYYY-MM-DD).                                  |53| `article_number`    | `int` or `null`  | Article number if applicable.                                               |54| `article_title`     | `str` or `null`  | Optional title of the article.                                              |55| `article_synthesis` | `str` or `null`  | Optional synthesis of the article.                                          |56| `text`              | `str` or `null`  | Text content of the explanatory_memorandum, article or file content (contenu du dossier) chunk.|57| `chunk_text`        | `str`            | Concatenated text (`title` + `article_text` or related content).            |58| `embeddings_bge-m3` | `str`            | Embedding vector of `chunk_text` using `BAAI/bge-m3`, stored as JSON string.|59 60---61 62## 🛠️ Data Processing Methodology63 64### 🧩 1. Content Extraction65 66Each **dossier législatif** was parsed, processed and standardized from his official XML structure. 67Metadata, article blocks, and explanatory sections were normalized into a unified schema. 68Specific rules applied per content type:69 70- `explanatory_memorandum`: Includes the explanatory's introduction only. All articles synthesis that are in the explanatory are split by their `article_number` and added to `article_synthesis`.71  Article fields are `null`.72  An explanatory memorandum (exposé des motifs) is an official text that accompanies a draft or proposed law.73  It is used to explain the reasons why the law is being proposed, the context in which it is set, and the objectives pursued by the legislator.74- `dossier_content`: Includes dossier's textual content if the split by article didn't work.75  The split may not work if there is no mention of article numbers in the dossier content or if the code was not adapted to a specific case in which the split wasn't possible.76  Article metadata fields are `null`.77- `article`: Structured content, where `article_number` and `text` are always present. `article_title` and `article_synthesis` may be missing.78 79- **Basic fields**: `doc_id` (cid), `category`, `title`, `number`, `wording`, `creation_date`, were taken directly from the source XML file.80- **Generated fields**:81  - `chunk_id`: A unique hash for each text chunk.82  - `chunk_index`: Indicates the order of a chunk within a same deliberation.83  - `chunk_xxh64`: is the xxh64 hash of the `chunk_text` value. It is useful to determine if the `chunk_text` value has changed from a version to another.84  - `content_type`: Nature of the content.85  - `article_number`: Number of the article. Available only if `content_type` is `article`.86  - `article_title`: Title of the article. Available only if `content_type` is `article`.87  - `article_synthesis`: Synthesis of the article extracted from the explanatory memorandum. Available only if `content_type` is `article`.88- **Textual fields**:89  - `text`: Chunk of the main text content.90    It can be an article text content extracted from the dossier content, a chunk of the explanatory memorandum's introduction or a chunk from the dossier content.91  - `chunk_text`: Combines `title` and the main `text` body to maximize embedding relevance.92    If `content_type` is `article`, then the article number is also added.93    94### ✂️ 2. Chunk Generation95 96A `chunk_text` was built by combining the `title`, the `article_number` if applicable and its corresponding `text` content section. 97Chunking ensures semantic granularity for embedding purposes.98 99No recursive split was necessary as legal articles and memos are inherently structured and relatively short for `content_type` = `article`. 100If needed, the Langchain's `RecursiveCharacterTextSplitter` function was used to make these chunks (`text` value). The parameters used are : 101 102- `chunk_size` = 8000103- `chunk_overlap` = 0104- `length_function` = len105 106---107 108### 🧠 3. Embeddings Generation109 110Each `chunk_text` was embedded using the [**`BAAI/bge-m3`**](https://huggingface.co/BAAI/bge-m3) model. 111The resulting embedding vector is stored in the `embeddings_bge-m3` column as a **string**, but can easily be parsed back into a `list[float]` or NumPy array.112 113## 🎓 Tutorials114 115### 🔄 1. The chunking doesn't fit your use case?116 117If you need to reconstitute the original, un-chunked dataset, you can follow [this tutorial notebook available on our GitHub repository](https://github.com/etalab-ia/mediatech/blob/main/docs/reconstruct_vector_database.ipynb).118 119⚠️ The tutorial is only relevant for datasets that were chunked **without overlap**.120 121### 🤖 2. How to load MediaTech's datasets from Hugging Face and use them in a RAG pipeline ?122 123To learn how to load MediaTech's datasets from Hugging Face and integrate them into a Retrieval-Augmented Generation (RAG) pipeline, check out our [step-by-step RAG tutorial available on our GitHub repository !](https://github.com/etalab-ia/mediatech/blob/main/docs/hugging_face_rag_tutorial.ipynb)124 125### 📌 3. Embedding Use Notice126 127⚠️ The `embeddings_bge-m3` column is stored as a **stringified list** of floats (e.g., `"[-0.03062629,-0.017049594,...]"`). 128To use it as a vector, you need to parse it into a list of floats or NumPy array. 129 130#### Using the `datasets` library:131 132```python133import pandas as pd134import json135from datasets import load_dataset136# The Pyarrow library must be installed in your Python environment for this example. By doing => pip install pyarrow137 138dataset = load_dataset("AgentPublic/dole")139df = pd.DataFrame(dataset['train'])140df["embeddings_bge-m3"] = df["embeddings_bge-m3"].apply(json.loads)141```142#### Using downloaded local Parquet files:143 144```python145import pandas as pd146import json147# The Pyarrow library must be installed in your Python environment for this example. By doing => pip install pyarrow148 149df = pd.read_parquet(path="dole-latest/") # Assuming that all parquet files are located into this folder150df["embeddings_bge-m3"] = df["embeddings_bge-m3"].apply(json.loads)151```152 153You can then use the dataframe as you wish, such as by inserting the data from the dataframe into the vector database of your choice.154 155---156 157## 🐱 GitHub repository : 158The project MediaTech is open source ! You are free to contribute or see the complete code used to build the dataset by checking the [GitHub repository](https://github.com/etalab-ia/mediatech)159 160## 📚 Source & License161 162### 🔗 Source:163- [**DILA** open data repository](https://echanges.dila.gouv.fr/OPENDATA/DOLE)164- [Data.gouv.fr : DOLE : les dossiers législatifs ](https://www.data.gouv.fr/datasets/dole-les-dossiers-legislatifs/)165 166### 📄 License:167**Open License (Etalab)** — This dataset is publicly available and reusable under the Etalab open license.