augustinian-babylm/synthetic-grounding-images

Synthetic grounding images

3,162 images generated to extend visual grounding to concrete words that no photograph dataset covers, for Augustinian BabyLM (paper, code).

How they were made

Starting from 1,986 concrete words with no image support, an LLM (claude-sonnet-4-6, temperature 0.8) wrote short scene descriptions placing as many target words as fit naturally into one scene. Each of the 1,054 resulting descriptions was rendered three times with SDXL-Turbo (2 inference steps, guidance scale 0, generator seeds 1000/1001/1002). Targets were then localized with OWLv2 at score threshold 0.25, and SAM features pooled inside the detected boxes; words the detector never found were dropped.

The scene descriptions and the LLM response cache are committed in the GitHub repository, so the whole set regenerates deterministically without API calls.

Layout

• —images/<id>_v<seed>.png — three variants per description
• —index.jsonl — one line per image: file, desc_id, and matched, the target words the description was written to contain

Caveats

Generated images and automatic detection are noisier than human region annotations, and the per-word benefit of synthetic grounding is roughly half that of real grounding. Detection near the score threshold is not perfectly stable across runs, so the set of successfully grounded words can vary by a few items between regenerations.

The generator declined to place a small number of anatomical and taboo words in scenes, so the coverage gap is not random: it is systematically the vocabulary a safety-tuned model avoids.

License

CC BY 4.0. Please cite the paper above.