Datasetpublic

edithatogo/fyi-archive-nz

fyi-archive-nz Derived FOI-O candidate layer Any separately versioned FOI-O/NLP output is an inferred candidate layer for research evaluation. It is not a gold label set, a certified legal finding, legal advice, or a replacement for the immutable raw archive. Candidate publication requires a separately approved dataset identity and target. Dataset card for incremental, read-only preservation of fyi.org.nz — the New Zealand Official Information Act (OIA) request… See the full description on the dataset page: https://huggingface.co/datasets/edithatogo/fyi-archive-nz.

sourceHugging Faceotherupdated 12h agoView on Hugging Face

0likes897downloads

Dataset Card

fyi-archive-nz

Derived FOI-O candidate layer

Any separately versioned FOI-O/NLP output is an inferred candidate layer for research evaluation. It is not a gold label set, a certified legal finding, legal advice, or a replacement for the immutable raw archive. Candidate publication requires a separately approved dataset identity and target.

Dataset card for incremental, read-only preservation of [fyi.org.nz](https://fyi.org.nz/) — the New Zealand Official Information Act (OIA) request register, running on Alaveteli.

The archive is being built incrementally. Early verified snapshots are published to Hugging Face and draft-first Zenodo deposits while the historical backfill continues. The manifest records indexed requests. Its record count does not prove that every request page or attachment has been stored or can be restored from this repository. Raw coverage requires object inventories and verified restoration receipts.

Source provenance

—Upstream: https://fyi.org.nz/ (public read-only endpoints only; no auth).
—Capture tool: `fyi-cli` (bulk enumeration + faithful WARC/WACZ capture of request metadata, rendered pages, and attachments).
—Orchestration: `edithatogo/fyi-archive`.

Data fields (per-request record)

field	type	description
`request_id`	int	fyi.org.nz numeric request id
`url_title`	string	canonical slug (final path segment of the request URL)
`title`	string	request title
`authority`	string	public body the request was addressed to
`state`	string	described state (e.g. `successful`, `refused`, `waiting`)
`first_sent`	datetime	when the request was lodged
`last_updated`	datetime	upstream last-modified
`content_sha256`	string	SHA-256 of the captured JSON payload (content address)
`html_captured`	bool	whether a rendered HTML snapshot exists
`attachments`	list<struct>	name, url, content_type, size, sha256
`warc_record_ids`	list<string>	WARC record ids for this request

Loading

python

from huggingface_hub import hf_hub_download

manifest_path = hf_hub_download(
    repo_id="edithatogo/fyi-archive-nz",
    repo_type="dataset",
    filename="manifests/latest_manifest.json",
)

sql

-- DuckDB over the Hugging Face mirror
SELECT authority, state, count(*) AS n
FROM 'hf://datasets/edithatogo/fyi-archive-nz/manifests/latest_manifest.parquet'
GROUP BY 1, 2 ORDER BY n DESC;

Intended use

Public-interest and policy research, journalism, reproducible historical preservation, and government-transparency analysis. The archive is not published for model training or generative-AI input.

Limitations

—A point-in-time snapshot of a live site; the authoritative record is the live fyi.org.nz site. See `NOTICE.md`.
—Not affiliated with or endorsed by the operators of fyi.org.nz.
—In the current verified snapshot, attachments, attribution, first_seen, last_updated, and per-record license have no non-null values; state is also largely unavailable. These fields are retained in the contract but are not inferred.
—The repository contains static Croissant metadata, but the Hub Dataset Server's generated Croissant endpoint is not currently available.

Copyright & licensing

Archived content is already-published public material; the dataset asserts no new rights over it. Per-item license + attribution are recorded in the manifest where the source declares them. Government-released material is typically CC BY 4.0 under the NZGOAL framework or "no known rights"; Acts/Regulations/court decisions are outside copyright (Copyright Act 1994 s.27); Crown works otherwise attract Crown copyright (s.26, 100-year term per IPONZ). Private-author request text remains the requester's. The repository code is MIT. Full provisions and citations: `docs/copyright-and-licensing.md`.

Citation

bibtex

@dataset{mordaunt_fyi_archive,
  author    = {Dylan Mordaunt},
  title     = {{fyi-archive: a read-only full-site archive of fyi.org.nz}},
  year      = {2026},
  url       = {https://github.com/edithatogo/fyi-archive}
}

Verified snapshot metadata

—Manifest verification: verified
—Snapshot records: 33,217
—Generated at: 2026-09-23T18:23:29.226584+00:00
—Manifest SHA-256: 0823ad1214d60264770692d801eff8eba79ac109280616f91096ea0b3a0cd604

This metadata describes the manifest snapshot; it does not verify raw HTML or attachment completeness. Manifest records are not a count of fully archived requests.