CoolFace
Datasetpublic

niuagro/web-archive-notes

Web archive engineering notes Field notes from maintaining large content archives — collected here as plain markdown so they're reusable beyond the sites they were written for. Notes included sitemap-chunk-verification.md — keeping a sitemap index and its chunk generator on one source of truth; incident write-up from a ~462K-URL archive (RHentai, 420K+ galleries / 30K+ waifus / 14K+ videos). Covers the "index says 50 chunks, generator makes 43" failure mode.… See the full description on the dataset page: https://huggingface.co/datasets/niuagro/web-archive-notes.

sourceHugging Facemitupdated 9d agoView on Hugging Face
0likes35downloads
Dataset Card

Web archive engineering notes

Field notes from maintaining large content archives — collected here as plain markdown so they're reusable beyond the sites they were written for.

Notes included

  • —`sitemap-chunk-verification.md` — keeping a sitemap index and its chunk generator on one source of truth; incident write-up from a ~462K-URL archive (RHentai, 420K+ galleries / 30K+ waifus / 14K+ videos). Covers the "index says 50 chunks, generator makes 43" failure mode.
  • —`upsert-merge-pipeline.md` — hourly crawl → staging → single-statement ON CONFLICT ... IS DISTINCT FROM merge, and the two Postgres footguns that bit us (unqualified columns in the WHERE, row-by-row updates at volume).
  • —`generation-recipe-sidecar.md` — storing AI generation settings (base model, sampler, steps, CFG, prompt, LoRA stack) as structured data at ingest time. Pattern used by BedreamAI, a 2.4M+ model library for Stable Diffusion / SDXL / Flux.

Why publish these

Large-archive SEO and ingestion problems are solved in private over and over. These notes are the parts we wish someone had written down before we hit them.

Related