david-ar/deb-packages

Binary package checksums (Debian, Ubuntu, Kali)

Per-artifact checksums of binary packages (.deb/.udeb) as shipped, harvested from the distros' own apt Packages indexes. Every row carries the pool path, so the claim URL is constructible: archive base + Filename. Digest coverage varies by distro — each row is a single witness carrying every digest its index states:

• —debian — trixie, trixie-updates, trixie-security, sid (all four components + debian-installer udeb sub-indexes): md5 + sha256, plus the debian-installer images tree (initrds, kernels — SHA256SUMS per versioned release)
• —ubuntu — every release ever indexed with SHA256 (gutsy 2007 → resolute 26.04, plus regenerated -updates/-security indexes back to warty 2004): release + -updates + -security pockets, all four components (main, restricted, universe, multiverse), enumerated from archive.ubuntu.com (supported) and old-releases.ubuntu.com (EOL — pool included, so claim URLs stay live for dead releases): md5 + sha1 + sha256 + sha512 — 4-hash single-witness rows
• —kali — kali-rolling: md5 + sha1 + sha256

2,890,481 rows across 129 witnesses (debian 152,570 / ubuntu 2,665,512 / kali 72,399), architecture amd64 (arch:all packages included), harvested 2026-08-14.

Layout

• —data/debs.parquet — witness, sha256, sha1, md5, sha512, name, version, arch, filename; sorted by sha256. filename is the pool path relative to the archive root (deb.debian.org/debian, security.debian.org/debian-security, archive.ubuntu.com/ubuntu, old-releases.ubuntu.com/ubuntu for EOL Ubuntu codenames, or http.kali.org/kali, by witness).
• —maps/sha1.parquet — sha1 → sha256 (ubuntu/kali rows), sorted by sha1.
• —maps/md5.parquet — md5 → sha256 (all witnesses), sorted by md5.
• —debs.sha256.bloom — DCSO-format membership filter over the 1,675,146 distinct sha256 keys (p = 10⁻⁴, lowercase-hex keys). hdx filters fetch debs installs it so hdx scan can route matching files to this dataset.
• —debs.tsv.gz — the normalized harvest the parquet is built from.

Both parquet files carry page indexes (zstd, no dictionaries, 128 KiB pages), so a point lookup over HTTP range reads fetches a few tens of KB. This is how hashdex resolves hashes from the CLI and from a static browser page — and how hdx recipe attaches fetchable claim URLs to the pool debs inside a Debian install image (a verified, generalized jigdo).

Rebuild

tools/deb_packages.py (index GETs only, never content) then tools/debs_parquet.py, both in this repo and in the hashdex GitHub repo. Suites are a point-in-time snapshot; re-harvest refreshes them.