AgentPublic/data-gouv-datasets-catalog

📢 Sondage 2026 : Utilisation des datasets publiques de MediaTech

Vous utilisez ce dataset ou d’autres datasets de notre collection MediaTech ? Votre avis compte ! Aidez-nous à améliorer nos datasets publiques en répondant à ce sondage rapide (5 min) : 👉 https://grist.numerique.gouv.fr/o/albert/forms/gF4hLaq9VvUog6c5aVDuMw/11 Merci pour votre contribution ! 🙌

🇫🇷 Data.gouv.fr Datasets Catalog

This dataset contains a processed and embedded version of the catalog of datasets published on [data.gouv.fr](https://www.data.gouv.fr), the French open data platform. The dataset was published by data.gouv.fr on the dedicated dataset page.

It includes rich metadata about each public dataset: title, URL, publisher organization, description, tags, licensing, update frequency, usage metrics, and more. The dataset provides semantic-ready and structured for semantic indexing and retrieval.

Each chunk has been embedded using the BAAI/bge-m3 model, making this catalog ready for search, classification, or retrieval-augmented generation (RAG) pipelines.

🗂️ Dataset Contents

The dataset is provided in Parquet format and contains the following columns:

🛠️ Data Processing Methodology

📥 1. Field Extraction

The original dataset was retrieved directly from the official dedicated dataset page. This dataset only includes data.gouv.fr datasets that have at least a 100 characters description to remove as much noise as possible from incomplete datasets.

✂️ 2. Text Chunking

The chunk_text field was created by combining the title, organization name, description.

The description was here cropped to a maximum length of +- 1000 characters. The Langchain's RecursiveCharacterTextSplitter function was used to crop the description.

The parameters used are :

• —chunk_size = 1000 (in order to limit as much noise as possible)
• —length_function = len

Then, only the first splitted text was keeped. Which leads to have a cropped description of a maximum of +- 1000 characters.

🧠 3. Embedding Generation

Each chunk_text was embedded using the **`BAAI/bge-m3`** model. The resulting embedding vector is stored in the embeddings_bge-m3 column as a string, but can easily be parsed back into a list[float] or NumPy array.

🎓 Tutorials

🔄 1. The chunking doesn't fit your use case?

If you need to reconstitute the original, un-chunked dataset, you can follow this tutorial notebook available on our GitHub repository.

⚠️ The tutorial is only relevant for datasets that were chunked without overlap.

🤖 2. How to load MediaTech's datasets from Hugging Face and use them in a RAG pipeline ?

To learn how to load MediaTech's datasets from Hugging Face and integrate them into a Retrieval-Augmented Generation (RAG) pipeline, check out our step-by-step RAG tutorial available on our GitHub repository !

📌 3. Embedding Use Notice

⚠️ The embeddings_bge-m3 column is stored as a stringified list of floats (e.g., "[-0.03062629,-0.017049594,...]"). To use it as a vector, you need to parse it into a list of floats or NumPy array.

Using the datasets library:

import pandas as pd
import json
from datasets import load_dataset
# The Pyarrow library must be installed in your Python environment for this example. By doing => pip install pyarrow

dataset = load_dataset("AgentPublic/data-gouv-datasets-catalog")
df = pd.DataFrame(dataset['train'])
df["embeddings_bge-m3"] = df["embeddings_bge-m3"].apply(json.loads)

Using downloaded local Parquet files:

import pandas as pd
import json
# The Pyarrow library must be installed in your Python environment for this example. By doing => pip install pyarrow

df = pd.read_parquet(path="data-gouv-datasets-catalog-latest/") # Assuming that all parquet files are located into this folder
df["embeddings_bge-m3"] = df["embeddings_bge-m3"].apply(json.loads)

You can then use the dataframe as you wish, such as by inserting the data from the dataframe into the vector database of your choice.

🐱 GitHub repository :

The project MediaTech is open source ! You are free to contribute or see the complete code used to build the dataset by checking the GitHub repository

📚 Source & License

🔗 Source :

• —Data.gouv.fr : Catalogue des données de data.gouv.fr

📄 Licence :

Open License (Etalab) — This dataset is publicly available and can be reused under the conditions of the Etalab open license.