lappland22/famine-letters

Irish Famine Letter Activity Dashboard

Per-letter analysis of Irish Famine relief correspondence (Nov 1845 – Feb 1849). 5,115 letters segmented from 3,857 OCR'd pages across 6 parliamentary volumes, with metadata extracted by Qwen3.5-27B-FP8.

Source corpus

• —Papers relating to Proceedings for Relief of Distress — Series 4, 5–7, 8 (Jun 1847 – Feb 1849; Poor Law Union perspective)
• —Correspondence — Commissariat Series — 184511–184607, 184607–184701, 184701–184703 (Nov 1845 – Mar 1847; Treasury / Commissariat perspective)

Tabs

• —📍 Activity Map — county/union × time animated map of letter volume
• —✍️ Writers — leaderboard of 679 writers (after canonical normalization); drill into per-writer activity timeline and top unions addressed
• —🗺️ Districts — county-level rankings + county × month heatmap
• —📂 Letter Browser — full filterable index of all 5,115 letters by sender, recipient, union, type, date, subject

Data files

• —letter_index.csv — 5,115 letters: sender, recipient, date, union, county, type, subject (no body)
• —letter_stats.csv — union × month aggregations
• —writer_stats.csv — per-writer aggregations (nletters, topunions, top_recipients, time range)
• —district_stats.csv — county × month aggregations
• —series_summary.csv — per-volume summaries
• —dim_writer.csv — 679 writers with all aliases collapsed (e.g. "Mr. Trevelyan", "C. E. Trevelyan", "Charles Trevelyan" → one writer_id)
• —plu_coordinates.csv / county_coordinates.csv — geo lookup

Pipeline

1. 1.Stage 01 — MinerU OCR → page-level markdown
2. 2.Stage 02 — LLM page classification (letter / dissolution / table / circular / etc.)
3. 3.Stage 05 — Qwen3.5-27B VLM repair of OCR errors
4. 4.Stage 06 — Two-stage letter segmentation:
5. 5.Phase 1: 5-page sliding window LLM emits letter boundaries (header/signature anchors only)
6. 6.Phase 2: Python deterministic body slicing using anchors
7. 7.Phase 3: per-letter LLM extracts structured metadata in batches
8. 8.Stage 06d — Writer canonical normalization (1,527 unique strings → 679 writer_ids)

Built using Qwen3.5-27B-FP8 + Streamlit + Plotly.