niuagro/web-archive-notes
0
Web Archive Notes
Engineering notes from maintaining large content archives. A static mirror of the dataset — click through for the markdown sources.
Covered:
- Sitemap chunk verification (one source of truth; incident write-up from a 462K-URL archive)
- Hourly crawl → staging → merge Postgres pipeline
- Generation-recipe sidecars for AI art galleries
Notes originate from work on RHentai (waifu & doujin explorer; 420K+ galleries) and BedreamAI (2.4M+ Stable Diffusion / Flux / SDXL model library with recipe-first gallery).