lszoszk/treaty-bodies-general-comments

Treaty Bodies General Comments

A paragraph-level dataset of General Comments and General Recommendations adopted by the nine UN human-rights Treaty Bodies, with concerned-group labels and document metadata. Companion to the UNHRD search interface.

Licence

The curated dataset (paragraph segmentation, label annotation, document metadata enrichment, footnote and section work) is released under **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)**.

You are free to:

• —Share — copy and redistribute the material in any medium or format
• —Adapt — remix, transform, and build upon the material

Under the following terms:

• —Attribution — give appropriate credit (see Citation below), provide a link to the licence, and indicate if changes were made.
• —NonCommercial — you may not use the material for commercial purposes without prior written permission from the authors.
• —ShareAlike — if you remix, transform, or build upon the material, you must distribute your contributions under the same licence.

The underlying General Comments and General Recommendations are issued by United Nations Treaty Bodies and remain under the UN's content terms; the curation work — segmentation, labelling, metadata enrichment — is the licensable contribution to which CC BY-NC-SA 4.0 applies.

For commercial licensing enquiries: <l.szoszkiewicz@amu.edu.pl>.

Dataset Summary

This is the v3.1 package (3 September 2026) built from the live UNHRD corpus (https://lszoszk.github.io/generalcomments/). Each row is a paragraph from a UN Treaty Body General Comment / General Recommendation, with section path, footnotes, footnote cross-reference resolutions (Ibid./op. cit./"see paragraph N above"), preamble flag, concerned-group labels, per-paragraph treaty-article references, and document-level metadata.

What's new in v3.1 vs. the v3 package (May 2026):

• —Footnotes recovered. The footnote layer was re-derived from the OHCHR DOCX/PDF editions where the earlier extraction had dropped notes or numbered them by Word's internal id rather than the printed number. 3,203 footnote entries (was 2,976); n is now always the number as printed. Two resolved_text annotations in E/C.12/2002/11 were withdrawn because their target note changed (53 resolved, was 55).
• —Six signatures corrected — A/41/38→A/41/45, A/45/81→A/45/18, A/87/18→A/8718 (×3), A/90/18→A/9018 (pre-1976 GA numbering). document_id / paragraph_id are unchanged.
• —`source_url` now points at UN Documents (docs.un.org/en/<symbol>) instead of the retired OHCHR download handlers.
• —`cited_articles` gained 6 references from a manual audit pass (source: "opus-audit"); 4,619 references across 2,365 paragraphs.

What's new in v3 vs. the v2 package:

• —`cited_articles` — a new per-paragraph column. Each entry resolves an "article N" mention in the paragraph text to a specific treaty (e.g. an "article 6" reference inside a CRC General Comment is tagged {treaty: "ICCPR", article: "6", source: "sonnet"} when the context makes clear it refers to the ICCPR, not the CRC). 4,613 references across 2,365 paragraphs. Resolution provenance is recorded in source: regex (home-treaty default, 3,989), sonnet (LLM disambiguation, 374), opus (LLM tie-breaker, 250). Cross-treaty, joint-GC and non-treaty references (UDHR, UN Charter, regional conventions → treaty: "?") are handled explicitly. See Data Fields → v3 enrichment below.
• —Refreshed against the current corpus: paragraph re-splitting and metadata-noise cleanup since v2.

Current package statistics:

• —187 GC source documents
• —7,216 paragraph-level segments
• —5,094 segments with at least one concerned-group label
• —3,203 footnote entries across 1,838 paragraphs
• —4,619 treaty-article references across 2,365 paragraphs
• —33 preamble paragraphs (is_preamble: true)
• —6,346 paragraphs with section path
• —19 distinct concerned-group labels

Treaty bodies represented:

• —CAT, CAT-OP, CCPR, CED, CEDAW, CERD, CESCR, CMW, CRC, CRPD

Supported Tasks

• —Multi-label classification of concerned-group labels in treaty body text segments
• —Semantic search and retrieval over segmented general comments
• —Weak supervision, label enrichment, taxonomy alignment work on human-rights text

Languages

English (the source UN documents in this package are English-language editions).

Dataset Structure

Single train split stored as parquet. Supplemental files:

• —document_index.parquet — one row per source document with title, signature, committee metadata, adoption date, source URL, and document-level label coverage statistics
• —GC_info.json — bundled source metadata used to enrich the dataset
• —scripts/prepare_treaty_bodies_hf.py — preparation script

Data Instances

{
  "row_id": "cat-op-gc-1::1",
  "paragraph_id": "cat-op-gc-1-0001",
  "document_id": "cat-op-gc-1",
  "document_title": "General comment No. 1 (2024) on places of deprivation of liberty (article 4)",
  "document_title_short": "SPT GC1: places of deprivation of liberty (Art. 4)",
  "signature": "CAT/OP/GC/1",
  "adoption_date": "11 February 2024",
  "adoption_year": 2024,
  "committee": "CAT-OP",
  "committees": ["CAT-OP"],
  "source_url": "https://docs.un.org/en/CAT/OP/GC/1",
  "is_joint_document": false,
  "footnotes_verified": true,
  "footnotes_source": "ohchr-pdf-no-fn-zone",
  "segment_position": 1,
  "paragraph_number": 1,
  "section": ["I. Introduction"],
  "is_preamble": false,
  "text": "The Subcommittee considers that to advance the prevention…",
  "labels": ["Persons deprived of their liberty"],
  "footnotes": [
    {
      "n": 2,
      "text": "Optional Protocol, preamble.",
      "is_ibid": false,
      "is_cross_ref": false,
      "is_self_ref": false,
      "resolved_text": ""
    },
    {
      "n": 3,
      "text": "Ibid., art. 1.",
      "is_ibid": true,
      "is_cross_ref": false,
      "is_self_ref": false,
      "resolved_text": "Optional Protocol, preamble."
    }
  ],
  "footnote_count": 3,
  "cited_articles": [
    {
      "treaty": "ICCPR",
      "article": "9",
      "paragraph": null,
      "confidence": "high",
      "source": "sonnet"
    }
  ],
  "cited_article_count": 1
}

Data Fields

Identity

• —row_id — synthetic stable identifier <document_id>::<segment_position>
• —paragraph_id — UNHRD canonical id (e.g. crc-c-gc-25-0042)
• —document_id — UNHRD doc slug (e.g. crc-c-gc-25)

Document metadata (denormalised on every row)

• —document_title, document_title_short, signature, ohchr_symbol, source_url
• —adoption_date, adoption_year
• —committee, committees — single-string + list (joint comments carry both committee codes)
• —is_joint_document — boolean
• —alternative_ids — additional docId slugs (some joint comments are reachable via two docIds)
• —articles — convention articles cited in the front matter
• —languages_available — ISO codes for available official translations
• —status — final or revision marker
• —first_added_at, last_verified_at — ISO date strings
• —footnotes_verified — boolean (true on 187/187 docs)
• —footnotes_source — provenance of the footnote layer (pdf-extraction-pipeline, ohchr-docx, user-verified, hri-compilation-pdf, ohchr-pdf-no-fn-zone, verified-by-pattern-analogy)

Paragraph fields

• —segment_position — 1-based position within the document
• —paragraph_number — official ¶ number (may differ from segment position when the doc has unnumbered preamble entries; preamble rows are paragraph_number: 0)
• —text — normalised paragraph text
• —text_length_chars, text_length_words — length stats
• —labels — zero or more concerned-group labels
• —label_count, has_labels — label coverage flags

v2 enrichment

• —section — list of strings, the section hierarchy this paragraph sits under (root → leaf). Empty list for documents without sections.
• —is_preamble — true for the 33 paragraphs that sit outside the numbered sequence (resolution-style preambles)
• —preamble_source — provenance of the preamble extraction
• —footnotes — list of struct entries:
• —n — footnote number as printed in the source
• —text — footnote body
• —is_ibid, is_cross_ref, is_self_ref — boolean flags
• —references_note, references_para — when set, point at the referenced footnote / paragraph for cross-references
• —resolved_text — for ibid./op.cit./inline cross-references, the materialised text of the original target (so downstream pipelines can read "Optional Protocol, preamble." instead of "Ibid.")
• —footnote_count — convenience count of footnotes length

v3 enrichment

• —cited_articles — list of struct entries, one per resolved treaty-article reference in the paragraph text:
• —treaty — the treaty the article belongs to. A bundle code (ICCPR, ICCPR-OP1, ICCPR-OP2, ICESCR, ICESCR-OP, CRC, CRC-OPAC, CRC-OPSC, CRC-OPIC, CEDAW, CEDAW-OP, CERD, CRPD, CRPD-OP, CMW, CAT, OPCAT, CED) or one of two markers: "?" (the article belongs to a non-treaty instrument — UDHR, UN Charter, a regional convention — outside the bundle) and "AMBIGUOUS" (the article is cited equally to two or more treaties, e.g. "in the two Covenants", and cannot be resolved to one).
• —article — article number as a string (e.g. "6", "14").
• —paragraph — sub-paragraph number when the citation is specific (e.g. "article 14 (3)" → paragraph: "3"); otherwise null.
• —confidence — high / medium / low on LLM-resolved entries; null on regex entries.
• —source — resolution provenance: regex (home-treaty default), sonnet (LLM disambiguation pass), opus (LLM tie-breaker on ambiguous cases), opus-audit (references added or corrected in the August 2026 manual audit of the article-reference grammar).
• —cited_article_count — convenience count of cited_articles length

Dataset Creation

Source Data

The v2 package is built directly from the live UNHRD curated corpus (docs/corpus.json + docs/documents.json in the lszoszk/generalcomments repository). The corpus is the same data the dashboard at <https://lszoszk.github.io/generalcomments/> serves at search time.

Provenance per document is exposed in footnotes_source:

• —pdf-extraction-pipeline — extracted from OHCHR PDF via PyMuPDF
• —ohchr-docx — extracted from OHCHR DOCX (e.g. CEDAW GR39)
• —hri-compilation-pdf — recovered from HRI/GEN/1/Rev.9 Vol. I/II
• —user-verified — manually checked against the source by a curator
• —ohchr-pdf-no-fn-zone — PDF has no inline footnotes
• —verified-by-pattern-analogy — pattern matched against a sister GC

Processing

The HF v3.1 package is produced from _docs_internal/build_hf_dataset.py (in the source repo). The script:

1. 1.Reads docs/corpus.json (the unified UNHRD corpus) and docs/documents.json (per-document metadata)
2. 2.Filters to type == "gc" (this dataset covers General Comments only — JUR / SP previews are separate)
3. 3.Sorts paragraphs by (docId, idx) for stable parquet output
4. 4.Maps each paragraph onto the v3 schema (above), denormalising the document metadata onto every row
5. 5.Builds a per-document document_index.parquet with paragraph counts, footnote counts, and provenance
6. 6.Writes a dataset_metadata.json build summary

The cited_articles layer is produced by a separate three-stage pipeline before the build: a regex pass tags every "article N" mention with the issuing committee's home treaty; a Sonnet pass re-resolves paragraphs flagged ambiguous (joint GCs, cross-treaty references, non-treaty instrument mentions); an Opus pass acts as a tie-breaker on the medium/low-confidence and ambiguous cases. The source field on every reference records which stage resolved it.

Label Distribution

Top concerned-group labels in the v3.1 package (full distribution in dataset_metadata.json):

• —Children: 2,394
• —Women/girls: 1,719
• —Persons with disabilities: 801
• —Migrants: 667
• —Persons deprived of their liberty: 365
• —Indigenous peoples: 281
• —Persons affected by armed conflict: 268
• —Refugees & asylum-seekers: 259
• —Adolescents: 233
• —Persons living in rural/remote areas: 215

Labels and Annotation Process

Concerned-group labels are generated through rule-based keyword matching. The repository script labels_annotation.py defines a mapping from each label to a curated list of keywords and phrases, then assigns every label whose keyword list matches a paragraph's text. These labels are best understood as heuristic weak labels for search, filtering, and exploratory analysis — not exhaustive expert annotations.

Full label inventory (19):

Adolescents, Children, Children in alternative care, Indigenous peoples, Internally displaced persons, LGBTI+, Migrants, Non-citizens and stateless, Persons affected by armed conflict, Persons affected by natural disasters, Persons deprived of their liberty, Persons in street situations, Persons living in poverty, Persons living in rural/remote areas, Persons living with HIV/AIDS, Persons with disabilities, Refugees & asylum-seekers, Roma, Gypsies, Sinti and Travellers, Women/girls.

Considerations for Use

• —Labels are weak/heuristic. Concerned-group labels are generated through curated keyword matching (see labels_annotation.py in the source repository). False positives, false negatives, and missed contextual mentions are possible. Treat them as a starting point for faceted search and weak supervision, not as gold-standard expert annotations.
• —Labels are sparse on some documents. 2,122 / 7,216 paragraphs have no labels (29%). This includes most pre-2000 CCPR Comments and some procedural / definitional paragraphs.
• —Footnote provenance varies. All 187 documents are footnotes_verified: true, but the footnotes_source differs: some are PDF-extracted with strong OCR confidence, some recovered from the HRI compilation, some manually verified by a curator. For high-stakes uses, consult the footnotes_source field.
• —Cross-reference resolution is partial. A subset (53) of the 3,203 footnotes carry is_ibid / is_cross_ref / is_self_ref annotations with resolved_text. The remaining footnotes are presented verbatim from the source PDF and may include un-resolved "Ibid." or "op. cit." references that downstream pipelines should treat with care.
• —`cited_articles` is a mix of deterministic and LLM-resolved entries. 3,989 of the 4,619 references are source: "regex" — a deterministic home-treaty default that is reliable for single-committee GCs but may be wrong for cross-treaty mentions. 624 are source: "sonnet" / "opus" — LLM-resolved, carrying a confidence field — and 6 source: "opus-audit" come from a manual audit pass. References to non-treaty instruments are tagged treaty: "?" and references cited equally to multiple treaties treaty: "AMBIGUOUS"; filter these out if you need only bundle-resolvable citations.
• —Section paths are auto-extracted. Some documents (notably CERD GR31, CESCR GC13) have rich Roman+Letter+Arabic 3-level hierarchies that the auto-extractor handles ~95% correctly with occasional wrap artifacts on multi-line headings.
• —No predefined train/validation/test split. Users should split by document_id to avoid leakage if training models.
• —Source UN documents are not under this licence. The CC BY-NC-SA 4.0 covers the curation work; the underlying General Comments remain under United Nations content terms.

Citation

When citing the dataset:

Szoszkiewicz, Ł. & Kowalska, Z. (2026). UNHRD — Treaty Bodies General
Comments dataset (paragraph-level corpus with concerned-group labels).
https://huggingface.co/datasets/lszoszk/treaty-bodies-general-comments.
Licensed under CC BY-NC-SA 4.0.

@dataset{szoszkiewicz_kowalska_unhrd_2026,
  author    = {Szoszkiewicz, Łukasz and Kowalska, Zuzanna},
  title     = {UNHRD — Treaty Bodies General Comments dataset
               (paragraph-level corpus with concerned-group labels)},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/lszoszk/treaty-bodies-general-comments},
  license   = {CC BY-NC-SA 4.0}
}

When citing individual paragraphs in academic work, please reference the original UN document signature (e.g. CRC/C/GC/25 ¶12), not this dataset.

Companion Software

The interactive search dashboard built on this corpus is open-source under PolyForm Noncommercial 1.0.0 (Zenodo concept DOI 10.5281/zenodo.10495719):

• —Live: <https://lszoszk.github.io/generalcomments/>
• —Code: <https://github.com/lszoszk/generalcomments>

Direct Use

from datasets import load_dataset
import pandas as pd

# Paragraph-level dataset
ds = load_dataset(
    "lszoszk/treaty-bodies-general-comments",
    split="train",
)
print(ds)
# Dataset({
#   features: ['row_id', 'paragraph_id', 'document_id', ...,
#              'section', 'is_preamble', 'footnotes', 'footnote_count',
#              'cited_articles', 'cited_article_count'],
#   num_rows: 7216
# })

# Filter to paragraphs about a specific concerned group
children = ds.filter(lambda r: "Children" in r["labels"])
print(len(children), "child-related paragraphs")

# Document-level metadata
documents = pd.read_parquet(
    "hf://datasets/lszoszk/treaty-bodies-general-comments/document_index.parquet"
)
print(documents.shape)

Recipes

# Cross-references resolved (Ibid./op.cit./see para N)
xrefs = ds.filter(
    lambda r: any(
        f["is_ibid"] or f["is_cross_ref"] or f["is_self_ref"]
        for f in r["footnotes"]
    )
)

# Paragraphs in a specific section
women_in_conflict = ds.filter(
    lambda r: r["document_id"] == "cedaw-c-gc-30-add-1"
              and r["section"]
              and "Recovery" in r["section"][0]
)

# Cross-treaty references — paragraphs in a GC issued by one
# committee that cite an article of a *different* treaty
cross_treaty = ds.filter(
    lambda r: any(
        c["treaty"] not in ("?", "AMBIGUOUS")
        and c["treaty"] != r["committee"]
        for c in r["cited_articles"]
    )
)

# Every paragraph that cites ICCPR article 6 (right to life)
iccpr6 = ds.filter(
    lambda r: any(
        c["treaty"] == "ICCPR" and c["article"] == "6"
        for c in r["cited_articles"]
    )
)