ArthurSrz/open_codes

1---2license: etalab-2.03language:4  - fr5tags:6  - legal7  - french-law8  - embeddings9  - legifrance10  - mistral11size_categories:12  - 10K<n<100K13configs:14  - config_name: default15    data_files:16      - split: train17        path: data/train-*.parquet18  - config_name: jurisprudence19    data_files:20      - split: train21        path: jurisprudence/train-*.parquet22  - config_name: circulaires23    data_files:24      - split: train25        path: circulaires/train-*.parquet26  - config_name: reponses_legis27    data_files:28      - split: train29        path: reponses_legis/train-*.parquet30---31 32# Open Codes33 34Open dataset of French legal code articles with embeddings.35 36Chunked articles from French legal codes sourced from37[Legifrance](https://www.legifrance.gouv.fr/) via the PISTE API,38with 1024-dimensional embeddings generated by Mistral AI.39 40Each row is a text chunk enriched with full article metadata from the parent41legal code article.42 43## Dataset Description44 45- **Source**: [PISTE Legifrance API](https://piste.gouv.fr/) (official French government legal database)46- **License**: [Licence Ouverte / Etalab 2.0](https://www.etalab.gouv.fr/licence-ouverte-open-licence/)47- **Embeddings**: Mistral AI `mistral-embed` (1024 dimensions)48- **Update frequency**: Daily (nightly sync at 02:00 UTC, dataset push at 07:00 UTC)49- **Codes**: Dynamically sourced from `LEX_codes_piste` (active codes only)50- **Quality**: Dedup + stale-chunk filtering applied before every push51 52### Legal codes included53 54The list of legal codes is dynamic and managed in the `LEX_codes_piste` table.55Adding a new code there (with `actif=true`) automatically includes it in the next sync and export.56 57## Schema58 59### Chunk fields60| Column | Type | Description |61|--------|------|-------------|62| `chunk_text` | string | Text content of the chunk |63| `embedding` | float32[1024] | Mistral AI embedding vector |64| `id_legifrance` | string | Legifrance article identifier |65| `code_name` | string | Human-readable code name (e.g. "Code civil") |66| `chunk_index` | int32 | Chunk position within the article (0-indexed) |67| `start_position` | int32 | Character offset in original article text |68| `end_position` | int32 | End character offset in original article text |69| `code` | string | Legal code identifier (e.g. LEGITEXT000006070721) |70| `num` | string | Article number (e.g. "L. 1234-5") |71| `etat` | string | Article status (VIGUEUR, ABROGE, etc.) |72| `fullSectionsTitre` | string | Full hierarchy path in the code |73 74### Article metadata fields (prefixed `article_`)75 76#### Identifiers77| Column | Type | Description |78|--------|------|-------------|79| `article_id_legifrance` | string | Legifrance article ID |80| `article_code` | string | Legal code ID |81| `article_num` | string | Article number |82| `article_cid` | string | Consolidated ID |83| `article_idEli` | string | ELI (European Legislation Identifier) |84| `article_idEliAlias` | string | ELI alias |85| `article_idTexte` | string | Text ID |86| `article_cidTexte` | string | Consolidated text ID |87 88#### Content89| Column | Type | Description |90|--------|------|-------------|91| `article_texte` | string | Full article plain text |92| `article_texteHtml` | string | Full article HTML |93| `article_nota` | string | Article notes (plain text) |94| `article_notaHtml` | string | Article notes (HTML) |95| `article_surtitre` | string | Article subtitle |96| `article_historique` | string | Article history |97 98#### Dates & Status99 100**Date format**: `article_dateDebut` and `article_dateFin` are stored as **Unix timestamps in milliseconds** (string type). This is the raw format returned by the Legifrance PISTE API.101 102To convert to a human-readable date in Python:103 104```python105from datetime import datetime, timezone106 107timestamp_ms = "1301529600000"108dt = datetime.fromtimestamp(int(timestamp_ms) / 1000, tz=timezone.utc)109print(dt)  # 2011-03-31 00:00:00+00:00110```111 112Special value: `32472144000000` (year 2999) means "no end date" — the article is in force indefinitely.113 114| Column | Type | Description |115|--------|------|-------------|116| `article_dateDebut` | string | Effective start date (Unix ms) |117| `article_dateFin` | string | Effective end date (Unix ms, `32472144000000` = indefinite) |118| `article_dateDebutExtension` | string | Extension start date (Unix ms) |119| `article_dateFinExtension` | string | Extension end date (Unix ms) |120| `article_etat` | string | Status: `VIGUEUR` (in force), `ABROGE` (repealed), etc. |121| `article_type_article` | string | Article type |122| `article_nature` | string | Legal nature |123| `article_origine` | string | Origin (e.g. `LEGI`) |124| `article_version_article` | string | Version identifier |125| `article_versionPrecedente` | string | Previous version ID |126| `article_multipleVersions` | bool | Has multiple versions |127 128#### Hierarchy129| Column | Type | Description |130|--------|------|-------------|131| `article_sectionParentId` | string | Parent section ID |132| `article_sectionParentCid` | string | Parent section consolidated ID |133| `article_sectionParentTitre` | string | Parent section title |134| `article_fullSectionsTitre` | string | Full hierarchy path |135| `article_ordre` | int32 | Sort order within the code |136| `article_partie` | string | Partie (e.g. "Partie legislative") |137| `article_livre` | string | Livre |138| `article_titre` | string | Titre |139| `article_chapitre` | string | Chapitre |140| `article_section` | string | Section |141| `article_sous_section` | string | Sous-section |142| `article_paragraphe` | string | Paragraphe |143 144#### Extras145| Column | Type | Description |146|--------|------|-------------|147| `article_infosComplementaires` | string | Additional info (plain text) |148| `article_infosComplementairesHtml` | string | Additional info (HTML) |149| `article_conditionDiffere` | string | Deferred condition |150| `article_infosRestructurationBranche` | string | Branch restructuring info |151| `article_infosRestructurationBrancheHtml` | string | Branch restructuring (HTML) |152| `article_renvoi` | string | Cross-references |153| `article_comporteLiensSP` | bool | Contains SP links |154| `article_idTechInjection` | string | Technical injection ID |155| `article_refInjection` | string | Injection reference |156| `article_numeroBo` | string | BO number |157| `article_inap` | string | INAP code |158 159## Configs160 161| Config | Description | Source |162|--------|-------------|--------|163| `default` | Code articles + chunks from Legifrance PISTE API | `REF_codes_legifrance` + `REF_article_chunks` |164| `jurisprudence` | Court decisions from Judilibre API | `REF_decisions_judilibre` + `REF_legal_chunks` |165| `circulaires` | Government circulars | `REF_circulaires` + `REF_legal_chunks` |166| `reponses_legis` | Parliamentary written answers | `REF_reponses_ministerial` + `REF_legal_chunks` |167 168## Usage169 170```python171from datasets import load_dataset172from datetime import datetime, timezone173 174# Load default config (code articles)175ds = load_dataset("ArthurSrz/open_codes", split="train")176 177# Load jurisprudence config (court decisions)178juris = load_dataset("ArthurSrz/open_codes", "jurisprudence", split="train")179 180# Access a chunk with its embedding and article metadata181row = ds[0]182print(row["code_name"])                # e.g. "Code civil"183print(row["chunk_text"][:200])184print(len(row["embedding"]))           # 1024185 186# Convert dates from Unix ms to datetime187date_debut = datetime.fromtimestamp(int(row["article_dateDebut"]) / 1000, tz=timezone.utc)188print(date_debut)                      # e.g. 2011-03-31 00:00:00+00:00189 190# Filter by legal code191code_civil = ds.filter(lambda x: x["code_name"] == "Code civil")192 193# Filter active articles only194en_vigueur = ds.filter(lambda x: x["article_etat"] == "VIGUEUR")195 196# Use embeddings for semantic search197import numpy as np198query_emb = np.array(ds[0]["embedding"])199```200 201 202 203## Provenance204 205Built by the [marIAnne](https://github.com/ArthurSrz/open_codes) project.206Sync pipeline fetches articles nightly from PISTE Legifrance, chunks them,207and generates embeddings via Mistral AI.208