niuagro/web-archive-notes
Web archive engineering notes Field notes from maintaining large content archives — collected here as plain markdown so they're reusable beyond the sites they were written for. Notes included sitemap-chunk-verification.md — keeping a sitemap index and its chunk generator on one source of truth; incident write-up from a ~462K-URL archive (RHentai, 420K+ galleries / 30K+ waifus / 14K+ videos). Covers the "index says 50 chunks, generator makes 43" failure mode.… See the full description on the dataset page: https://huggingface.co/datasets/niuagro/web-archive-notes.
Conversations for this repository live on Hugging Face.
CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.
Open discussions on Hugging Face