niuagro/web-archive-notes

Web archive engineering notes Field notes from maintaining large content archives — collected here as plain markdown so they're reusable beyond the sites they were written for. Notes included sitemap-chunk-verification.md — keeping a sitemap index and its chunk generator on one source of truth; incident write-up from a ~462K-URL archive (RHentai, 420K+ galleries / 30K+ waifus / 14K+ videos). Covers the "index says 50 chunks, generator makes 43" failure mode.… See the full description on the dataset page: https://huggingface.co/datasets/niuagro/web-archive-notes.