Chrisyichuan/screenshot-training-natural-filtered-4o-40k

Chrisyichuan/screenshot-training-natural-filtered-4o-40k

Re-filtered version of Chrisyichuan/screenshot-training-natural-filtered-v2 where hard-negative selection is done with GPT-4o (OpenAI) instead of the original GPT-4.1-mini filter.

GPT-4.1-mini was found to over-drop valid hard negatives at a 44% rate on a 110-query head-to-head sample (cases where its step-1 answer didn't match the GT). GPT-4o reduced this to ~22% and gpt-4o was more conservative and more precise overall. This dataset uses the new mining run.

Splits

• —train: 39482 rows (subset, first 40K of v2 train)
• —eval: 5704 rows (full v2 eval)
• —test: 5696 rows (full v2 test)

Row schema (*_hn_with_answer.jsonl)

{
  "query": "...",
  "chunk_path": "images/shard_xxx/.../chunk_0000_05.png",
  "neg_chunk_paths": [
    "images/shard_xxx/.../chunk_0000_03.png",
    "images/shard_xxx/.../chunk_0000_07.png"
  ],
  "source_positive_rank": 1,
  "source_positive_score": 0.609,
  "answer": "..."
}

Three jsonl variants per split:

• —<split>.jsonl — query + positive chunk_path only
• —<split>_hn.jsonl — adds neg_chunk_paths
• —<split>_hn_with_answer.jsonl — full schema (above)

Mining + filter pipeline

1. 1.Retrieval (`:30888`): top-20 candidates per query via FAISS IVFFlat over base Qwen3-VL-Embedding-2B (28M Wikipedia tile embeddings, dim=2048).
2. 2.VLM filter (`gpt-4o`): two-step (answer + judge) on each candidate; verdict CORRECT → drop as false negative, otherwise keep as HN.
3. 3.Selection: --candidate-k 10 --num-hard-negatives 2 early-stop, so exactly 2 HN per kept row.

Image storage

Images are packed as tar shards under image_shards/. To materialize:

python extract_hf_image_shards.py --dataset-dir .

Image Storage

The images are stored as 1000 tar shards under image_shards/ to keep the repository file count low and make uploads/downloads more reliable.

To materialize the images locally after download:

python extract_hf_image_shards.py --dataset-dir .