Datasetpublic

soyames/arwen-policy-corpus

Arwen Policy Corpus Canonical dataset: soyames/arwen-policy-corpus on Hugging Face Pipeline repository: soyames/arwen-policy on GitHub Provenance-preserving collection of processed policy documents from Internet governance and digital-policy sources. Quick Start from datasets import load_dataset ds = load_dataset("soyames/arwen-policy-corpus") print(ds["train"][0]["text"][:200]) print(ds["train"][0]["source_family"]) Corpus Contents Documents:… See the full description on the dataset page: https://huggingface.co/datasets/soyames/arwen-policy-corpus.

sourceHugging Faceapache-2.0updated 19d agoView on Hugging Face

1likes63downloads

Dataset Card

Arwen Policy Corpus

Canonical dataset: soyames/arwen-policy-corpus on Hugging Face Pipeline repository: soyames/arwen-policy on GitHub

Provenance-preserving collection of processed policy documents from Internet governance and digital-policy sources.

Quick Start

python

from datasets import load_dataset

ds = load_dataset("soyames/arwen-policy-corpus")
print(ds["train"][0]["text"][:200])
print(ds["train"][0]["source_family"])

Corpus Contents

—Documents: 192
—Total characters: 3,900,453
—Sources: 17 organizations
—Languages: primarily English

Source	Documents
ICANN	61
IETF	51
IGF	15
ITU	13
ARIN	7
EU	7
ISOC	7
APNIC	6
RIPE	6
Academic	5
UNESCO	5
UN	4
LACNIC	1
OECD	1
AFRINIC	1
IANA	1
other	1

Extraction Methods

—html_bs4: 187
—pypdf: 5

Languages

—en: 159
—en-US: 14
—und: 7
—en-us: 6
—en-AU: 5
—es: 1

Record Fields

Field	Type	Description
`document_id`	string	Unique document identifier
`source_id`	string	Content hash prefix
`source`	string	Organization name (ICANN, IETF, etc.)
`source_family`	string	Source organization family
`source_url`	string	Original URL
`final_url`	string	URL after redirects
`artifact_sha256`	string	Content hash
`content_type`	string	MIME type
`extraction_method`	string	html_bs4, pypdf, etc.
`byte_size`	int	Raw content size
`title`	string	Document title
`language`	string	BCP-47 language tag
`text_length`	int	Character count
`characters`	int	Character count (alias)
`text`	string	Full extracted text
`segments`	string	JSON array of text segments
`published_at`	string	Publication date (ISO 8601)
`date_source`	string	How the date was determined
`date_confidence`	string	Date extraction confidence
`date_type`	string	Date type classification
`no_date_reason`	string	Why date is missing (if applicable)
`retrieved_at`	string	When the document was captured
`http_status`	int	HTTP response code
`jurisdiction`	string	Geographic jurisdiction
`policy_topics`	string	JSON array of policy topics
`license`	string	Document license
`access_conditions`	string	Access restrictions
`date_evidence_json`	string	JSON with full date provenance

Provenance

Every record includes provenance metadata tracing the extraction chain: source URL -> HTTP capture -> content extraction -> segmentation -> normalization.

Date provenance tracks how each publication date was determined, with confidence levels (high, medium, low) and source classification.

Design Principles

—Provenance first: every derived record traceable to a source artifact
—Multistakeholder by construction: explicit modelling of stakeholder representation
—Absence is not neutrality: missing perspectives recorded without interpretation
—Candidate is not verified fact: automated extraction produces candidates requiring validation
—Reproducible releases: versioned metadata and processing history

Limitations

—Early-stage research corpus; automated extraction may introduce errors
—English-centric: most documents are in English
—Coverage is not uniform across all Internet governance organizations
—Dates are not available for all documents (118 of 192 undated)
—Segments are automatically generated and may not perfectly align with document structure

Source Code

https://github.com/soyames/arwen-policy

Citation

bibtex

@dataset{arwen_policy_corpus,
  title={Arwen Policy Corpus},
  author={Yao Amevi Amessinou Sossou},
  year={2026},
  url={https://huggingface.co/datasets/soyames/arwen-policy-corpus},
  license={Apache-2.0}
}