luc-prr904/ipid-corpus

IPID corpus acquisition layer

Phase 1 of a larger project: collect and catalogue European Insurance Product Information Documents (IPIDs) for affinity insurance lines, and produce a manifest that describes every document precisely enough to be useful six weeks later.

An IPID is the standardised two-page pre-contractual summary that EU insurers must publish for non-life products (IDD, Regulation (EU) 2017/1469). They are public, free, and follow a common template across the 27 member states. The UK kept the document after Brexit as an FCA requirement — those are collected too and tagged regime = "FCA" rather than "IDD".

Scope of this phase. A local corpus of PDFs plus a manifest describing each one, reproducible from scratch with one command and resumable if it dies halfway. There is no field extraction, no parsing of insurance terms, no LLM reading of document contents and no graph construction here. Those depend on this layer being clean and come later.

Lines in scope: mobile, appliance, gadget, travel, bike, ppi. Markets in scope: FR, BE, DE, ES, IT, NL, UK.

Architecture

Discovery is agentic and unreliable; fetching is deterministic and reliable. The whole design keeps them apart. No LLM ever touches bytes.

STAGE A: DISCOVERY   (agentic, smolagents + DeepSeek via the HF router)
   input:  seed targets (insurer/distributor × country × product line)
   output: candidate URLs written to DuckDB table `candidates`
   the agent may be wrong, may hallucinate URLs, may return nothing — that is fine

STAGE B: FETCH       (deterministic, no LLM anywhere)
   input:  `candidates` rows with status = 'pending'
   output: PDFs on disk + rows in `documents`
   this stage must be boring, idempotent, and fully resumable

STAGE C: CATALOGUE   (deterministic, no LLM anywhere)
   input:  PDFs on disk
   output: enriched `documents` rows — hash, page count, language, IPID-validity check

Each stage is a subcommand and can be re-run independently. Stage A was built last, on purpose: with B and C already trustworthy, the agent's output can be judged on its own rather than debugged simultaneously with the fetcher.

Install and run

Requires Python 3.11+ and uv.

uv sync --extra discovery

--extra discovery pulls in smolagents and ddgs, which the default search tool runs on. Stages B and C do not need either, and do not need a token — you can fetch and catalogue on a machine with no model access at all. Discovery itself needs HF_TOKEN for the model and no search key: it searches through DuckDuckGo by default.

cp .env.example .env   # then set a real contact email; HF_TOKEN only for Stage A

.env is read automatically by every ipid command (a shell variable of the same name still wins, so HF_TOKEN=… ipid discover works for a one-off). .env is gitignored.

IPID_CONTACT_EMAIL and IPID_CONTACT_URL are published in the User-Agent of every request this project makes — that is their only purpose:

ipid-corpus/0.1 (+https://example.com/ipid; ops@example.com)

Every insurer whose site you read sees that line in their logs. It is the difference between a crawler an operator can ask about and one they can only block, so it wants to be a real, monitored inbox that someone answers — not a personal address, and not the example.org placeholder. ipid prints a warning to stderr while it is still the placeholder.

uv run ipid discover --country FR --line mobile --limit 50
uv run ipid fetch --limit 100
uv run ipid catalogue
uv run ipid status

Two more subcommands sit outside the three-stage contract:

• —ipid import-candidates <file> loads hand-written candidate URLs (YAML or JSONL of {url, entity, country, line}, optionally entity_id and a registry identifier) straight into candidates. It exists so the fetcher can be exercised and reviewed without spending a token on discovery.
• —ipid reset --blocked returns candidates parked on an unreadable robots.txt to the queue. ipid fetch already does this for transient failures on its own; the explicit form, with --no-transient-only, is for deciding to ask again a host that said no.

Unattended runs: ipid campaign

For a night's work rather than a single command. A campaign plan lists steps in order; the campaign runs them in small batches of discover → fetch → catalogue until the plan is finished, the budget is spent, or the deadline arrives.

ipid campaign config/campaigns/france-night.yaml --dry-run   # what would run, what is skipped and why
ipid campaign config/campaigns/france-night.yaml             # run it

Leave the terminal open, or run it with nohup ... &, and keep a laptop on power: on macOS the campaign holds an idle-sleep assertion (caffeinate) for as long as it runs, but a closed lid on battery still sleeps.

What it does about the failures seen in supervised runs:

data/campaigns/<name>/report.md is rewritten after every batch, so it is readable mid-run: new documents with their IPID verdicts, each batch's outcome per combination, and every pause and failure. Child-process logs are in logs/campaigns/<campaign-id>/.

It does not fix bugs. It contains them — isolates, retries, skips — and writes them down.

Without uv

python3 -m venv .venv && .venv/bin/pip install -e ".[discovery]" && .venv/bin/pip install pytest pytest-asyncio

Tests

uv run pytest

401 tests, ~65 seconds. The LLM is scripted in every test and the only network traffic is to a http.server running on localhost: no test costs money or needs a network.

Two live search checks are skipped by default. They cost nothing (search queries, not model calls) but do leave the machine:

IPID_LIVE_SEARCH=1 uv run pytest tests/test_search_live.py -v

What each stage does

Stage A — discovery

Starts from config/seeds.yaml (59 curated entries across the seven markets), not from open-ended crawling. For each (seed × line) the agent gets six tools and at most 12 steps:

The document is usually not on the seed's domain

This is the correction that reshaped Stage A. Affinity cover is underwritten by an insurer and administered by a platform, and the IPID is very often published by one of those rather than by the brand that sells it — Air France's travel IPID lives on magroup-online.com.

The old web_search injected site:<seed domain> into every query and discarded anything off-domain. That is not a conservative setting, it is a structural blind spot: no amount of searching can reach a document the filter forbids. Open-web search is now the primary tool, and finding the host is treated as half the job. site_search keeps the scoped behaviour for when the host is already known.

A search result naming this entity makes its host readable — a real widening of scope, recorded in the run log as hosts_opened rather than done silently.

Guardrails, all enforced in the tools rather than the prompt:

• —What the agent may read is set by `discovery.scope`. open (the default) means any public website: an insurer's, a white-label platform's, a comparison site's. Documents for affinity products are routinely published on hosts the brand never links to, so a narrower rule is a blind spot rather than a precaution. seed restores the one-hop rule: the seed's domain plus hosts it links documents to.
• —Open does not mean anything. The agent reads URLs out of arbitrary web pages, and a page can point it at http://192.168.1.1/, localhost or a cloud metadata address. Local names, private and link-local addresses, and public-looking names that resolve to them (10.0.0.1.nip.io) are refused in both scopes, before any request, and on every redirect hop. The seed's own domain is exempt: an operator who seeds a local server did it on purpose. Not covered: a name that resolves publicly at check time and privately at fetch time (DNS rebinding) — that needs connect-time address pinning, which the browser does not do.
• —Redirects are walked hop by hop, each one re-checked against robots.txt and the scope rule, so a redirect cannot carry the agent onto a host nobody vetted.
• —submit_candidate rejects a submission whose entity, country or line disagrees with the seed instead of silently correcting it, so drift shows up in the logs. On an aggregator seed (see covers below) the entity is chosen from a closed list we authored.
• —A page read refused by a WAF is reported as blocked_by_waf, never as a robots decision. The two need different responses, and conflating them misled the first diagnosis of this codebase.
• —`submit_candidate` refuses a document for another market. Language does not settle market — French is spoken in four of them — and one run filed SNCF Connect's Swiss and Luxembourg IPIDs (magroup-online.com/VSC/CH/FR/, /VSC/LU/FR/) as French: real IPIDs, in French, passing every other check. White-label platforms put the market in the path, so ipid.markets reads it from there, built for precision rather than recall: platform paths (/VSC/CH/FR/), locale tokens (fr-ch, it-it, AMD_FR_fr) and lone segments that can only be a country (/ch/, /at/). A lone /fr/, /de/ or /it/ is never read as a market, and neither is the hostname — a Swiss insurer can serve French customers from a .ch domain. The French overseas departments count as France. Measured on all 45 URLs in the manifest: 9 state a market, 7 match their seed, and the 2 conflicts are exactly the SNCF documents. ipid status lists any source already filed under the wrong market; it lists them and changes nothing.
• —`submit_candidate` refuses a product line we do not collect. The corpus is six affinity lines sold through a partner, not insurance in general; an insurer's own motor, household, health or life documents are out of scope even on the right domain. Of the 64 AXA documents a URL index returned, most were motor and home. An in-scope line term always wins, so /auto-moto-velo/ipid-velo.pdf is kept — it is a bicycle IPID whatever the folder is called.
• —Every step — thought, tool call, truncated observation, error — is appended to logs/discovery/{run_id}/{seed}--{line}.jsonl.
• —Token counts come from smolagents' own monitor and are converted to euros with the prices in config/settings.yaml; cost per document is measured, not estimated.
• —Zero candidates for a seed is a valid outcome, recorded as outcome = 'no_candidates'. There is no retry-with-a-different-prompt path; that would just be prompt-fishing.

What a run costs

deepseek-ai/DeepSeek-V4-Flash is served over the router by Novita, Fireworks AI and DeepInfra (Featherless AI was listed but erroring when checked on 2026-08-14), and Hugging Face bills provider rates through with no markup. The same model therefore costs different amounts depending on who answers, so every rate is listed and the assumption in force is recorded on the run:

pricing.provider: auto (the default) bills at the dearest provider, so a reported figure is an upper bound. pricing.usd_to_eur is your accounting rate — nothing fetches it. Both, plus the resulting per-token rates, are written into runs.params_json, because a cost_eur read back in six months means nothing without the rate that produced it.

Measured, not estimated. The first live run — one seed × line, AXA France / mobile — used 102,350 input and 2,106 output tokens and cost €0.0137. Input dominates completely, because smolagents resends the whole transcript each step and fetch_page observations are large; a run that goes the full 12 steps is therefore near the top of that range.

Extrapolating: the full seed file of 123 combinations is roughly €1.70, not the €0.60 a naive 30k-token-per-combination estimate suggests. HF gives free accounts $0.10 of credit a month and PRO accounts $2.00.

Two levers if that matters: lower max_steps, or trim fetch_page_char_limit and the 200-link cap in fetch_page, which is where most of those input tokens come from.

Search backends

discovery.search_backend in config/settings.yaml:

DuckDuckGo, measured on 2026-09-15. A first batch of ten agent-shaped queries at 3 s spacing: eight answered with ten hits each and exactly the documents Stage A had been failing to find — Air France's IPID on magroup-online.com, ITA Airways' on chubb.com, eight Jet2holidays IPIDs from one query. The two empty answers were not real: repeated after a 30 s pause they returned 7 and 10 hits. Twenty minutes and some forty queries later every engine returned nothing for a query that had just worked; two minutes after that, DuckDuckGo answered again with ten hits. So: usable, but intermittently throttled, and it recovers by itself within minutes.

Three things about ddgs shaped the backend, each read in its source or measured:

• —Its `auto` mode is not what it sounds like. It always spends its first two workers on wikipedia and grokipedia, which never have an IPID, before trying anything else. The engine is pinned with ddgs_backend: "duckduckgo".
• —It hides throttling. A scraped engine's HTTP 429 — measured on search.brave.com — comes back as No results found., and a challenge page parses as zero results. No exception tells us we are being throttled, so a run of empty answers is the only signal there is.
• —Its empty answers are sometimes spurious, so each one is retried once after 10 s before it is believed.

Which is why the scrapers cool down rather than give up: after three consecutive empty answers, search pauses for 180 s and then tries again, and it stops for the run only after three cool-downs. The old behaviour — off for the rest of the run after five empties — threw away search for hours of a long run over a two-minute throttle.

The agent waits out a cool-down rather than being told about it. The first version announced the pause to the model as "unavailable — do not call it again". In the first live run with it, the model re-asked on 7 of its 12 steps regardless, so a two-minute throttle cost a whole seed. Now the search call itself sleeps until the cool-down ends and then queries: the agent's search is slow instead of failed. Steps cost tokens; a batch run's wall-clock time costs nothing. Only a permanent stop reaches the model as "unavailable", and the wording no longer claims a pause is permanent.

Failures are never empty results. That was the root of the worst misdiagnosis in this project, so it is enforced at every layer: a rejected key, a malformed request, a persistent rate limit or a network failure raises a SearchError; "nothing found" is an empty list. A fatal error — rejected key, malformed request — stops discovery before the next seed instead of paying for agents that cannot search.

Search is checked before anything is spent. A discovery run sends one real query before it records a run or builds a model. A missing key or a refused preflight is one readable line from the CLI, costs nothing, and leaves no half-finished run in the manifest.

Two details from earlier runs still matter:

• —What counts as empty is what the caller could use, not what the engine returned. Bing answers a site: query with ten links from other domains. Counting the raw hits let a run report healthy search while the agent received nothing, every time.
• —The agent is told when search is unavailable. It used to receive [], indistinguishable from "no results for those words", and dutifully rephrased: one logged run issued 87 queries, of which 82 never reached the network. The tool now raises and names the tools that still work — during a cool-down as well.

runs.search_json records the query count, raw and usable hits, errors, cool-downs and why search stopped, if it did.

If you have a Brave key, search_backend: brave is the reliable option. smolagents' own ApiWebSearchTool is refused on purpose: it calls requests with no timeout, turns a rejected key into an exception indistinguishable from "no results", and round-trips its results through markdown, which loses any title containing ]. BraveSearchTool keeps its endpoint and auth header and fixes all three, and honours Brave's Retry-After and X-RateLimit-Reset — but refuses to sleep through a monthly quota. Note that it only understands Brave's response format; pointed at another provider it would return nothing.

Stage B — fetch

The stage everything else rests on, so it is deliberately dull.

• —robots.txt is fetched once per host per run (concurrent workers wait on one fetch rather than stampeding), parsed with urllib.robotparser, and honoured. Disallowed URLs are marked blocked_by_robots and never requested. There is no override flag.
• —Per-host token bucket, minimum 2 s between requests to the same host. Different hosts proceed in parallel; the same host never does. Global cap of 4 in-flight requests.
• —Redirects are followed by hand, checking robots.txt at every hop (max 5).
• —Retries: exponential backoff with full jitter, max 3 attempts, only on timeouts, connection errors, 429 and 5xx. 404 and 403 are never retried. 60 s total per URL. Retry-After on 429/503 is honoured exactly, and the whole host is held back for that period, not just the one worker.
• —Validation before storing: HTTP 200; %PDF- magic bytes (trusted over a lying Content-Type, in both directions); 10 KB–20 MB; opens in pypdf; 1–8 pages. Failing any of the first four is a hard reject and leaves no document row. Failing the page count is a soft reject — the manifest row survives with rejection_reason = 'page_count', and the file is filed under data/rejected/ instead of the corpus. Stage C then looks inside it; see below.
• —One request per URL, not per candidate. A white-label PDF is routinely the published document for several distributors and product lines, which since v3 is several candidate rows. They are grouped by URL and fetched once. distinct_urls in the run summary is what was actually asked for; attempted counts rows.
• —A robots.txt we could not read is not a refusal. robots_unreachable_* and robots_status_5xx return the candidate to pending on the next run, up to max_attempts. disallowed_by_robots and robots_forbidden_403 are answers and stay permanent — ipid reset --blocked --no-transient-only is the deliberate human override.
• —Content-addressed storage at data/raw/{sha256[:2]}/{sha256}.pdf, written to a temp file and renamed so a crash cannot leave a truncated PDF at a valid content address.
• —Dedup is a first-class outcome. The same IPID on an insurer site and a distributor site is one file and two document_sources rows. That relationship is exactly what a later phase wants as a graph edge, so it is recorded, not suppressed.
• —Idempotent and resumable. Statuses are pending → in_flight → fetched | failed_terminal | blocked_by_robots | rejected. in_flight rows older than 10 minutes are reset to pending at startup. The manifest, never the filesystem, is the completion signal. Running ipid fetch twice in a row makes zero network requests the second time — there is a test for it.

Stage C — catalogue

For every accepted PDF: sha256, bytes, page_count, pdf_producer, pdf_creation_date, plus

• —text_layer_present — under 200 extracted characters the PDF is a scan. It is flagged, not OCR'd; the scanned proportion per market is itself a finding.
• —Glyph-named text is decoded first. Some PDFs ship /uni004A/uni0061/uni0068… instead of a ToUnicode map, and pypdf extracts those names literally. Left alone the text is long enough to pass the scan threshold but detects as the wrong language and matches no headers — a row that looks catalogued and is worthless. One real document in the corpus went from English, 0/6 headers, not an IPID to German, 6/6 headers, IPID once decoded.
• —detected_language + confidence via lingua, restricted to the seven expected languages. Compared against the market's expected language(s); a mismatch is flagged, not treated as an error. Belgium legitimately accepts nl/fr/de.
• —looks_like_ipid — heuristic only, no LLM: an IPID keyword and at least two of the six standard section headers (what is insured / what is not insured / restrictions / obligations / where am I covered / when and how do I pay). The matched-header count is stored as an integer too.
• —format_variant — the Dutch Verzekeringskaart is a national standard that is IPID-adjacent but not identical. Those are collected and tagged, and a later phase can decide whether they are comparable.

IPIDs inside larger documents

Italy does not publish standalone IPIDs. It publishes a set informativo: a 19–53 page bundle whose first pages are the IPID and whose remainder is the full policy. French retailers do the same thing — Darty's 12-page file is IPID + FICA + pre-contractual notice. Judged as whole files these all fail the 8-page rule, and the entire Italian market read as empty.

Stage B is still right to reject them: the file is not an IPID. So Stage C reads the kept bytes page by page and looks for the shortest window of at most max_pages pages that passes the same test a standalone IPID has to pass — an IPID keyword plus at least two section headers. When it finds one it records ipid_page_first / ipid_page_last, and language, headers and format variant are measured on that window rather than on the forty pages of policy wrapped around it.

A document with a page range set counts as corpus; the file keeps its rejection_reason = 'page_count', which remains a true statement about the file. A bundle with no qualifying window stays out — 300 pages of policy wording that merely mentions the word is not an IPID.

On the real corpus this recovered 6 files for 0 new HTTP requests, because keep_soft_rejected had already put the bytes on disk:

Italy went from 0 documents to 5; the corpus went from 15 to 21. The last row is why this is a window search and not a "read the first two pages" rule.

bundle_scan_pages (default 80) bounds how far in Stage C will look — the search is quadratic in pages, and a 900-page policy wording is not hiding an IPID on page 400.

ipid status

One command that prints the state of the world. Output below is from a local fixture run, not real data:

Accepted documents — country × line
┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━┳━━━━━━━┓
┃ country ┃ mobile ┃ appliance ┃ gadget ┃ travel ┃ bike ┃ ppi ┃ total ┃
┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━╇━━━━━━━┩
│ DE      │      — │         — │      — │      — │    1 │   — │     1 │
│ FR      │      1 │         1 │      — │      — │    — │   — │     2 │
│ NL      │      — │         — │      — │      — │    1 │   — │     1 │
│ all     │      1 │         1 │      0 │      0 │    2 │   0 │     4 │
└─────────┴────────┴───────────┴────────┴────────┴──────┴─────┴───────┘
 distinct documents per cell; a shared document counts in each cell

Sources by publisher role
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ role                    ┃ source URLs ┃ documents ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ insurer                 │           3 │         3 │
│ distributor             │           1 │         1 │
│ published under >1 role │           — │         1 │
└─────────────────────────┴─────────────┴───────────┘

Candidates by status      Rejections and failures, ranked
┏━━━━━━━━━━━━━━━━━┳━━━┓   ┏━━━━━━━━━━━━━━━━━┳━━━┓
┃ status          ┃ n ┃   ┃ reason          ┃ n ┃
┡━━━━━━━━━━━━━━━━━╇━━━┩   ┡━━━━━━━━━━━━━━━━━╇━━━┩
│ fetched         │ 4 │   │ pdf: page_count │ 1 │
│ failed_terminal │ 1 │   │ fetch: http_404 │ 1 │
│ rejected        │ 1 │   └─────────────────┴───┘
│ total           │ 6 │
└─────────────────┴───┘

Corpus
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ metric                                   ┃        value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ source URLs stored                       │            5 │
│ unique documents behind them             │            4 │
│ dedup rate                               │        20.0% │
│ in corpus (file on disk)                 │            3 │
│   of which found inside a larger bundle  │            0 │
│ catalogued                               │            3 │
│ looks_like_ipid                          │            3 │
│ all six IPID sections found              │            1 │
│ T&Cs kept for phase 2                    │            1 │
│ format variants (e.g. verzekeringskaart) │            1 │
│ runs recorded                            │            2 │
│ tokens in / out                          │ 48210 / 6120 │
│ total spend                              │      €0.0175 │
│ cost per accepted document               │      €0.0058 │
└──────────────────────────────────────────┴──────────────┘

Two further tables are printed and omitted here: "Machine-readable vs scanned, by country" (the fixture corpus has no scans) and "Identity and search health", which reports how many entities still lack a registry identifier and how many discovery runs hit a search engine that was blocking us or ignoring site: — the two things easiest to leave undone and most expensive to discover late.

Layout

config/
  seeds.yaml            59 curated entities across FR BE DE ES IT NL UK
  ipid_keywords.yaml    multilingual keywords, section headers, anchor hints
  settings.yaml         politeness, validation thresholds, model id, prices
data/
  raw/{sha256[:2]}/{sha256}.pdf        the corpus
  rejected/{sha256[:2]}/{sha256}.pdf   soft rejects (T&Cs), kept for phase 2
  manifest.duckdb
logs/
  discovery/{run_id}/{seed}--{line}.jsonl
  fetch/{run_id}.jsonl
src/ipid/
  db.py fetch.py catalogue.py status.py validate.py retry.py robots.py
  ratelimit.py storage.py keywords.py urls.py models.py settings.py cli.py
  discover/  browsing.py search.py tools.py runner.py
tests/

Manifest schema

Every table carries a schema_version column, and the version is also stored in a meta table — ipid status warns when the file predates the code.

• —`entities` — entity_id (PK), entity, role, country, lei, orias, fca_frn. One row per legal entity; the registry columns are where the identity decision lands.
• —`candidates` — (url_normalised, entity_id, line) (PK), url, entity, role, country, sourcerunid, agentconfidence, agentrationale, status, discoveredat, claimedat, attempts, last_error
• —`documents` — sha256 (PK), storedpath, bytes, pagecount, textlayerpresent, detectedlanguage, languageconfidence, lookslikeipid, headermatchcount, matched_headers, formatvariant, **ipidpagefirst**, **ipidpagelast**, regime, firstfetchedat, cataloguedat, pdfproducer, pdfcreationdate, rejectionreason
• —`document_sources` — (sha256, url_normalised, entity_id, line) (PK), url, entity, role, country, fetched_at (many-to-one with documents; this is where dedup information lives)
• —`runs` — runid, stage, startedat, endedat, paramsjson, tokensin, tokensout, costeur, candidatesfound, documentsfetched, **searchjson**
• —`seed_runs` — per (run × entity_id × country × line) outcome, steps, candidates, tokens, cost, error

Beyond the spec: claimed_at on candidates (crash recovery needs to know when a row went in-flight), catalogued_at on documents (so Stage C can be re-run without redoing finished work), the seed_runs table (per-seed cost and the no_candidates outcome need somewhere to live; runs holds the aggregate), and the three v2 columns below.

Additive schema changes are one ALTER TABLE … ADD COLUMN IF NOT EXISTS line in db.MIGRATIONS; they run on every open, so a manifest written by an older version gains the new columns instead of failing on the first insert. A key change cannot work that way — DuckDB has no ALTER … ADD PRIMARY KEY — so v3 rebuilds the three affected tables from the new DDL, copying across by column name. It is tested against the verbatim v2 DDL rather than against the current schema with bits removed, and it ran on the real manifest without losing a row.

What v2 added, and why

`role` on `candidates` and `document_sources` — insurer | distributor | broker | mga, carried from the seed at submission time (never asked of the model). Before this, the retailer/insurer distinction lived only in seeds.yaml and could only be recovered by joining on a free-text company name. Now the Phase 3 edge is a query:

-- the same IPID published by both a retailer and the insurer behind it
SELECT sha256, list(DISTINCT entity), list(DISTINCT role)
FROM document_sources GROUP BY sha256
HAVING count(DISTINCT role) > 1;

`matched_headers` on `documents` — the six section slugs that were actually found (what_is_insured, what_is_not_insured, restrictions, obligations, where_am_i_covered, when_and_how_do_i_pay), not just how many. header_match_count tells you a document looks right; this tells a later phase which section to read for a given question, and which ones the extractor mangled:

SELECT count(*) FROM documents WHERE list_contains(matched_headers, 'restrictions');

Soft-rejected documents keep their bytes, under data/rejected/ rather than data/raw/. An over-length PDF is nearly always the full terms & conditions — not corpus, but an IPID that says "see clause 4.2" is unusable without it, and re-fetching later means crawling every insurer a second time. The corpus directory stays exactly the corpus, and ipid status counts them separately. Set validation.keep_soft_rejected: false to go back to deleting them. Keeping them turned out to matter more than expected: v3's bundle scan recovered six documents from bytes v2 had already saved, at no crawling cost.

What v3 added, and why

`entity_id`, and an `entities` table. entity was free text and rows were keyed by it, so "AXA France IARD" and "AXA France" were two companies and nothing could say otherwise. entity_id is authored in seeds.yaml — derived identity is exactly the problem — and is what candidates, sources and seed runs are keyed by. entities carries the name, role, country and the lei / orias / fca_frn columns, so the registry decision has somewhere to land rather than being deferred for want of a column. Registry values already filled in are never overwritten by a later run that does not carry them.

Candidates keyed by (URL, entity_id, line), and document_sources by (sha256, URL, entity_id, line). One URL is routinely the published document for two product lines, and one platform URL for several distributors; the old URL-only key kept whichever arrived first and dropped the rest in silence. Fetching still makes one request per URL.

`ipid_page_first` / `ipid_page_last` on `documents` — see IPIDs inside larger documents above. NULL means the file is the IPID.

`search_json` on `runs` — whether search was answering, stored next to what the run found.

`covers` on a seed makes it an aggregator. verzekeringskaarten.nl hosts every Dutch insurer's Verzekeringskaart; without this every card would be filed under the portal's name, which is worse than not collecting them. covers is a closed list of entities the agent may attribute a candidate to — it chooses between names we authored, it never supplies one.

- entity: "Verzekeringskaarten.nl"
  entity_id: verzekeringskaarten-nl
  role: distributor
  country: NL
  domain: verzekeringskaarten.nl
  lines: [travel, bike, gadget]
  covers:
    - {entity_id: unive, entity: "Univé", role: insurer, country: NL}
    - {entity_id: interpolis, entity: "Interpolis", role: insurer, country: NL}

What v4 added, and why

`channel` on entities — retailer | bank | airline | telecom | travel | membership | employer | platform | direct. role says who an entity is in the contract; channel says through what kind of relationship the cover is sold. They are independent: Fnac Darty and Halifax are both distributor, but a shop and a bank are not the same market, and affinity distribution is the axis this corpus is actually about. direct is a real answer, not a missing one — an insurer selling its own book is out of scope by design.

ipid status reports how much of the corpus is affinity rather than direct, because a corpus about affinity that is mostly direct has quietly stopped being the thing it claims.

Phase 2 — the target schema

Phase 1 records who published a document. That is not the same question as who carries the risk, and in affinity insurance the two almost never coincide: a retailer sells the cover, an MGA designs it, an insurer underwrites it, a third party administers the claims. Phase 1's entity is only ever the first of those, asserted by the seed.

The fields Phase 2 has to read out of the PDF text, so the intent is written down before the code exists:

Two of these are already partly in reach. The document date is often in pdf_creation_date. The source URL and country are in document_sources, several per document, which is the dedup relationship Phase 3 wants as a graph edge.

The cross-check that follows — comparing an IPID against its policy wording and its product page to catch conflicting insurer names, stale PDFs and changed underwriting — is also already provisioned for: keep_soft_rejected means the policy wordings are on disk under data/rejected/, not thrown away.

Phase 2 is built: ipid extract, ipid evaluate, ipid export

ipid extract    # read parties, product, identifiers, dates and sections out of each IPID
ipid evaluate   # score the insurer against hand-labelled gold (config/gold/)
ipid export     # data/exports/: products, relations, sections, and DATASHEET.md

Deterministic, no language model: the IPID template fixes where each fact sits — a "Company / Product" header and nine standard questions — and the rules read it in eight languages. Every value is stored with the verbatim quote, the page and the method it came from, and rows are keyed by EXTRACTOR_VERSION, so a changed rule adds rows beside the old ones instead of rewriting them.

The template's "Company:" is the product's manufacturer, which may be a broker that designed it (Fnac España names SPB Ibérica, a correduría). So the header's company and the insurer are separate fields, and the company becomes the insurer only on evidence: a known insurer, an explicitly insurer-worded label ("Versicherer:", "Onderneming (Verzekeraar):"), the document's own "X, schadeverzekeraar", or "a trading name of" X.

Measured on 2026-09-18, insurer at legal-entity level: 45/45 on a stratified gold set that includes the hard cases (it was used for tuning, so an upper bound), and 29/30 on a held-out set labelled blind (precision 1.0, recall 0.967; the miss is a header extracted without spaces). No insurer was invented for the three documents that name none. Only the insurer is scored; the datasheet says so for every other field.

Known limitations

Judgement calls and things that are honestly not solved.

Stage A works mechanically and produced nothing useful on its first live run. Both halves of that sentence matter.

The plumbing is fine: DeepSeek-V4-Flash does support tool calling through the router, the agent chose sensible French-language queries, every step was logged, and tokens and cost were accounted correctly. On its very first search it found exactly the right page — axa.fr/bibliotheque-ipid.html, AXA's IPID library.

Then it got nothing, for the environmental reasons documented below: its searches came back empty as DuckDuckGo's throttle bit, and list_pdf_links on the correct page returned [] because that library is a JavaScript search app. The agent spent its remaining steps re-searching and re-reading the same page, hit max_steps, and recorded no_candidates — the honest outcome, but 102k tokens for zero documents.

Across seven live combinations in three markets, yield was 3 candidates from 7 seeds, at a total cost of about €0.15. All three came from the Netherlands, all three were real, and all three came from link-walking rather than search. The four failures were sites whose documents are not in their HTML. The fix is not a better prompt: it is a search backend that answers, and accepting that SPA document libraries need a headless browser the stack deliberately does not have.

The cost figures rest on three assumptions. DeepInfra's rate came from its own model page; Novita's and Fireworks' came from a third-party aggregator, not the providers themselves. Which provider actually serves a request depends on your router settings, so auto deliberately bills at the dearest of them. And usd_to_eur is a constant in a config file — nobody fetches a rate. All three are recorded per run, so a wrong assumption can be corrected after the fact rather than silently baked into a number.

Corpus size. The 300–600 document target depends entirely on discovery yield, which is unmeasured for the reason above. Stages B and C impose no ceiling.

Free search works, but only intermittently. The history, because it explains the design:

• —August: smolagents' WebSearchTool answered its first query well, then DuckDuckGo served a bot challenge (HTTP 202 with a CAPTCHA) after a handful of queries, to our honest User-Agent and a browser one alike. bing kept answering but ignored site:.
• —September: DuckDuckGoSearchTool, which runs on ddgs and a different request path, answered eight of ten agent-shaped queries with ten relevant hits each and no rate-limiting; its two empties were spurious and recovered on repeat. After roughly forty queries in half an hour every engine went quiet for a couple of minutes, then DuckDuckGo answered again.

So the default is usable, not dependable. A long run will hit throttled stretches; the cool-down turns each one into a few minutes of link-walking rather than a dead run, but yield during those minutes comes from page reads alone. search_backend: brave with a key is the dependable option. Throttling is also shared by everything on the same address: probing search by hand before a run makes the run's search worse.

What was fixed is the agent's blindness to it. The empty-streak detector now counts usable results rather than raw ones — so bing's off-domain answers trip it — and the tool raises instead of returning [], so the model stops asking. In the logged run that behaviour cost 87 queries and 12 wasted steps per seed.

But search turned out to matter less than expected. In the first seven live combinations every single web_search returned empty — the throttle was already in force — and three of those seven still produced candidates. The Dutch seeds succeeded purely by link-walking: homepage → product page → the PDF, three fetch_page calls and done. What separated success from failure was not search but site structure: Centraal Beheer, Univé and Interpolis link to their documents from server-rendered product pages, while Wertgarantie, ERGO, assona and AXA do not expose them in HTML at all. Fixing search will raise yield; fixing it will not rescue a seed whose documents only exist behind JavaScript.

Sitemaps index pages, not documents. Measured against six real seed domains, two to three requests each:

Zero PDFs across 6,078 sitemap URLs. Insurers list their HTML pages and leave documents out, which is ordinary SEO practice. So list_sitemap_pdfs is not the document source this README first claimed it was: its real value is narrower and still real — on axa.fr it returns, in two requests and with no search engine, the exact page (bibliotheque-ipid.html) that the first live run needed a working search backend to find. It replaces search as a way to reach the right page; the documents still have to come off that page, and where the page is a JavaScript app they still do not.

Treat it as a cheap first move that sometimes hands list_pdf_links its target, not as a way to enumerate a corpus.

Some document libraries are genuinely JavaScript-rendered. AXA's own IPID library page (axa.fr/bibliotheque-ipid.html) serves 126 KB of HTML with a correct title, 27 links — and not one occurrence of the string .pdf. The document list arrives in a later XHR; even a real browser running JS showed 0 PDF links, because it is an interactive search app. list_pdf_links now also reads <iframe>, <embed>, <object>, data-* attributes and PDF paths embedded in inline scripts, which rescues the far commoner case of a page shipping its document list as embedded JSON — but there was nothing in AXA's page source to find, and there still is not. For that page the sitemap channel and the search backend are the only routes in. No fix within a no-headless-browser stack.

The default search backend impersonates a browser — for search-engine queries only. ddgs sends its requests through primp, an HTTP client whose purpose is to present a browser's TLS fingerprint and headers. That is a larger departure from this project's "httpx everywhere, no browser impersonation" rule than smolagents' Mozilla/5.0 header was, and it was taken knowingly, because it is what makes free search answer at all. It never touches an insurer or retailer site: Stage B and every tool that reads a page use httpx with the honest, contactable User-Agent and a complete set of request headers. search_backend: brave removes the exception entirely, at the price of a key; httpx keeps the rule and gets a bot challenge.

WAFs block discovery but not fetching. intersport-rent.fr returns 403 to our User-Agent on HTML while serving its PDF quite happily. Sending complete headers may help; it will not always: measured after the change, intersport-rent.fr still returns 403 — now for its sitemap as well as its pages — while serving its PDF. The asymmetry is real and is now recorded as blocked_by_waf, so "this host refused us at the edge" is distinguishable from "robots.txt said no". We can retrieve documents we cannot discover, and the log says which case it is.

Entity identity is plumbed, not decided. entity_id is now the key everything joins on, authored in seeds.yaml, and the entities table has lei, orias and fca_frn columns waiting. Nobody has filled them in — ipid status reports how many entities lack any registry identifier, currently all of them. The plumbing means the decision can be made once and applied everywhere; it does not make the decision. Until it is made, entity_id values are our slugs and two records of the same company in different markets are still two rows.

A `Retry-After` longer than 30 s ends the attempt rather than being shortened. Waiting less than a server asked for is worse than giving up and coming back next run.

Magic bytes are only checked in the first 64 bytes. A PDF hidden behind a long HTML preamble is rejected as content_type. In practice such a response is not a PDF.

`bundle_scan_pages` is a guess. 80 pages bounds a quadratic window search. No file in the corpus so far needed more than 53, but a market that publishes 200-page bundles with the IPID at the back would need it raised, and the scan would get slow.

The Italian market does not publish standalone IPIDs, and the page-count rule excludes it almost entirely. Every Italian document collected so far — ITA Airways, TAP, Heymondo, ViaggiSicuri, Global Assistance — is a set informativo: the DIP, the DIP aggiuntivo and the full policy conditions bound into one PDF of 19 to 53 pages. All five were soft-rejected on page count. This is not a handful of awkward files; it is how the market publishes, so on current rules Italy would be near-absent from the corpus while France, Germany and the Netherlands fill up.

The documents are not lost — checking the ITA Airways bundle, page 1 carries the IPID keyword and three standard headers and page 2 carries the other three, with the conditions starting afterwards. The DIP is exactly the first two pages. A page-range check ("do any two consecutive pages contain a keyword and ≥2 headers?") would recover the entire market, where a flat page cap cannot. That is the single highest-value change to Stage B, and it is why keeping soft-rejected bytes mattered: all five are on disk and can be re-catalogued without re-crawling.

Distributors bundle the IPID into a larger PDF, and the page-count rule rejects it. The first live fetch caught one: Darty's mobile insurance document is a 12-page file containing the IPID plus the FICA notice and the AEI terms. It is soft-rejected as page_count, correctly by the letter of the rule — but it does contain a real IPID, and its first page is unmistakably one. Expect this to be systematic among retailers rather than rare. Because soft rejects now keep their bytes, those documents are sitting in data/rejected/ and a later phase can extract the IPID pages without re-crawling; that decision paid for itself on the first run. If the pattern turns out to be common, the better fix is a page-range check for IPID section headers rather than a flat page cap.

`looks_like_ipid` is deliberately loose. Two of six section headers, not six: two-column PDF extraction routinely mangles a heading or two. Expect false positives on documents that quote the IPID structure, and check the header_match_count column before treating the flag as ground truth.

Language detection is restricted to seven languages (plus Catalan). On two pages of insurance boilerplate an unrestricted detector guesses badly. A document in a language outside that set will be misreported rather than reported as unknown.

Scanned PDFs are counted, not read. No OCR in this phase, by design.

Verzekeringskaart documents are collected, not judged. They are tagged format_variant = 'verzekeringskaart' and left for a later phase to accept or drop.

`pytz` is a dependency only because duckdb needs it to hand TIMESTAMPTZ values back to Python. The connection is pinned to UTC so two people reading one manifest see the same timestamps.

The seed list is hand-written from public knowledge of these markets. Every domain is a real company operating in that market and selling those lines, but no entry has been verified to actually publish an IPID at a discoverable URL — that is what Stage A is for. The role/line assignments are judgement calls and are cheap to correct: it is one YAML file.