yiyangd/pointarena_dataset

Molmo2 PointArena SFT Data

26,596 supervised pointing examples used to fine-tune Molmo2-8B (`yiyangd/molmo2-8b-ft`) into a stronger PointArena solver (76.2% → up from 73.9% base, +2.3 pp).

Provenance

Each record is (image, query, answer):

• —image: A LAION-2B image sampled by reservoir sampling. Stored under laion_images/<bucket>/<hash>.jpg (bucket is the first 2 hex chars of the SHA-1 hash of the image URL, used to spread files across folders).
• —query: A natural-language pointing query derived from the LAION caption (e.g. "Where is X?", "Point to X").
• —answer: A target point in Molmo2 format, <points coords="1 X Y">name</points>, where X and Y are integers in [0, 1000]. Generated by Molmo-7B-D-0924 as the teacher model.

Stats

How it was built

1. 1.Sample LAION captions/images uniformly (reservoir sampling).
2. 2.For each image, ask Molmo-7B-D to point at the LAION caption phrase using the prompt pointing: <phrase>.
3. 3.Parse outputs with the strict point format and keep only:
4. 4.valid <points coords="...">name</points> parse,
5. 5.coordinates inside [0, 1000],
6. 6.non-empty answer name.
7. 7.Convert to a single-turn chat record for Molmo2 SFT.

Use

import json
from datasets import load_dataset

ds = load_dataset("yiyangd/pointarena_dataset", split="train")
ex = ds[0]
# ex["image"] is the relative path, ex["query"] and ex["answer"] are strings

Or directly:

import json
for line in open("molmo2_train.jsonl"):
    rec = json.loads(line)
    img_path = rec["image"]   # e.g. "laion_images/5d/5dd2a6d23e667553.jpg"
    query    = rec["query"]   # e.g. "Where is the tim and eric's billion dollar movie?"
    answer   = rec["answer"]  # e.g. "<points coords=\"1 495 768\">...</points>"

Training recipe (LoRA SFT on Molmo2-8B)

• —base: allenai/Molmo2-8B
• —LoRA r=32, alpha=64
• —target modules: att_proj, attn_out, ff_proj, ff_out
• —per-device batch 1, grad accumulation 16 (effective batch 16)
• —lr 1e-5, warmup ratio 0.05, 1 epoch
• —bf16, single L40S (~3.5 h)

Trained checkpoint: `yiyangd/molmo2-8b-ft`.

License

• —Image sources: LAION-2B (CC-BY-4.0 derived; see LAION's terms).
• —Annotations: produced by Molmo-7B-D-0924 (Apache 2.0).
• —Combined dataset: released under CC-BY-4.0.

LAION images are publicly available URLs and are redistributed here under the same terms LAION uses. If you are a rights-holder for any image, contact the author to request removal.

Citation

@misc{molmo2-pointarena-ft,
  title  = {Molmo2-8B fine-tuned for PointArena (LAION + Molmo-7B-D distill)},
  author = {Yiyang Dai},
  year   = {2026},
  url    = {https://huggingface.co/yiyangd/molmo2-8b-ft}
}