domeist/text-auto-illustrate

Text Auto Illustrate — passage-to-image relevance judgements

Relevance judgements for illustrating prose: given a paragraph of Wikipedia text, which images from a 5.4-million-image collection actually suit it?

Two judgement sets over the same 25 passages — one annotated by hand, one generated and far broader — plus the metadata for every image either set names, so the benchmark can be used without downloading the underlying corpus.

Built for a University of Glasgow final-year project in 2022, because no ground truth for this task existed. Documented and released in 2026.

The task

Not image captioning, and not image search from a keyword query. The input is a whole paragraph of running prose, and the output is a ranked list of images that would sit well beside it. The difficulty is twofold: deciding what a 200-word passage is actually about, and matching that against a collection where the only text attached to each image is a Wikipedia caption.

Files

qrels_*.csv follow the usual TREC-style shape — one row per passage–image pair:

passage_id,image_id,relevance
0,0,1
0,2824798,1

Row counts are pairs, not images: an image judged relevant to two passages appears twice. The two sets name 722 and 83,332 distinct images respectively.

The two judgement sets

Strict — annotated by hand

25 passages, 722 distinct images, 28 per passage on average (ranging from 5 to 113). Someone read each passage, looked through candidate images, and decided.

This is the more trustworthy set and the smaller one. Its weakness is incompleteness: with 5.4 million images available, no human found every suitable one. In practice a retriever returns plainly appropriate images that score nothing because nobody judged them. A passage about fur-trade forts retrieving Hudson's Bay Company trading posts scores zero.

Treat strict scores as a lower bound.

Lenient — generated from article structure

25 passages, 83,332 distinct images, 4,421 per passage on average (763 to 22,833). Built on the assumption that images near a passage's own Wikipedia article are plausible illustrations for it:

Generated by walking each article's outgoing links with the wikipedia package and collecting the images found. Grade 1 is deliberately generous, so this set measures something considerably weaker than the strict one.

Treat lenient scores as an upper bound.

Read them together

Neither set is the truth. Strict understates quality because its judgements are incomplete; lenient overstates it because its judgements are loose. A system's real quality sits between them.

Loading it

from datasets import load_dataset

passages = load_dataset("domeist/text-auto-illustrate", "passages")["train"]
qrels    = load_dataset("domeist/text-auto-illustrate", "qrels_strict")["train"]
images   = load_dataset("domeist/text-auto-illustrate", "images")["train"]

print(passages[0]["text"][:80])
# In architecture, the frieze is the wide central section part of an entabla…

images.csv carries the id, caption, image URL and source article for every judged image, so a system can be evaluated against these judgements without the full corpus. To rank against the whole 5.4-million-image collection, the corpus is a [release asset on GitHub][release] (566 MB gzipped) or can be rebuilt from [WIT][wit].

[release]: https://github.com/domeist/text-auto-illustrate/releases/latest [wit]: https://github.com/google-research-datasets/wit

Image ids

Ids are positions in the corpus, assigned sequentially across WIT's English training split in sorted filename order. They mean nothing on their own — join against images.csv, or against a corpus built by [tools/format_wit.py][tools]. Converting a different subset of WIT, or the same files in a different order, shifts every id.

[tools]: https://github.com/domeist/text-auto-illustrate/blob/main/tools/format_wit.py

Reference results

BM25 over image captions and article titles, 10 query terms per passage, top 100 retrieved, measured over the full 5,411,977-image corpus.

Three things to know before comparing against these.

Lenient recall cannot approach 1. With ~4,400 images judged relevant per passage and only 100 retrieved, the ceiling is 100 ÷ 4,400 ≈ 0.047. A reported 0.006 is about an eighth of what is reachable.

Ties matter here. Captions are short, so many documents receive identical BM25 scores, and tied results are ordered arbitrarily. On a caption-only run, reordering ties alone moved Precision@5 between 0.120 and 0.152. Report how your system breaks ties.

The passage set is small. 25 passages means Precision@5 moves in steps of 0.008. Differences of a few hundredths are not results.

Provenance

Passages and captions are Wikipedia text. The images are Wikimedia Commons URLs; no image files are redistributed here, matching what [WIT][wit] itself distributes. Individual images carry their own licences, unaffected by this one.

The relevance judgements were produced for this project and are released under the same licence so the sets can be used whole.

Licence

CC BY-SA 3.0, inherited from Wikipedia via WIT. Share and adapt freely, with attribution, under the same licence.

Dataset: Srinivasan, Raman, Chen, Bendersky and Najork. WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning. SIGIR 2021.

Text and captions: Wikipedia contributors, CC BY-SA 3.0.

Code

Retrieval system, evaluation harness and the scripts that built these sets: github.com/domeist/text-auto-illustrate (code MIT, data CC BY-SA 3.0).