vineet-datasets/probe-benchmark-hard50

PROBE hard50 — human-reviewed hard development set

This is a separate standard-format review bank of 62 Astra-conditioned hard development episodes. It is NOT an independently evaluated holdout. Do not merge into or modify benchmark_600.

The original 50 human-reviewed questions were augmented with 12 human-accepted inverse comparison prompts, yielding 24 compare questions total.

Contents and ordering

bank_manifest.json lists exactly these 62 episodes, using schemaversion 1 and the same manifest/episode/review structure as `approved-questions/benchmark600`.

Each episode has review.json, after_walls_topdown.png, verification.png, GT overlays/metadata, selected_objects.json, gt_visibility.json, and scene_snapshot/scene_snapshot.json. Reference images live under _ref_views/; frozen table textures live under _texture_store/.

All review.status fields are pending. The hard_episode.selection_status field records model-failure mining acceptance, NOT human approval. Selection policies are heterogeneous and faithfully retained: paired failures, earlier user-authorized single failures, explicit red-cube 2-of-3 exception, and later at-least-2-of-3 trials (early stopping after two valid failures where authorized). Do not relabel all episodes as 2/3.

Full available accepted-attempt histories are retained inside attempt_1/, attempt_2/, and where applicable attempt_3/, including correct attempts. Robot witnesses and QA evidence are retained. Bank-level acceptance/recipe metadata is under each episode's provenance/bank/. Original review JSON is under provenance/source_review.json.

Website agent handoff

1. 1.Register approved-questions/benchmark-hard50 as a separate review bank using the existing scripts/review_server.py manifest/review loader. No website configuration or running server was changed during consolidation.
2. 2.Use each manifest ep_dir as the unique episode identity. The copied review's episode index matches this bank's 0–49 numbering. Source snapshots, question filenames and attempt files intentionally retain original internal episode numbers, as the official bank also does.
3. 3.Serve review.images relative to the episode directory; serve review.reference[*].images relative to the bank root. References are namespaced _ref_views/ep_NNN_type/... to avoid collisions between source banks. Do not reconstruct reference paths from object names.
4. 4.Keep review status pending until the human adjudicates. Preserve existing unrelated website banks and the user's review server.
5. 5.Transfer the entire bank for full evidence access, not just the PNGs. source_mapping.json maps all new IDs to original accepted banks. copy_checksums.json records byte-verified copies. validation.json records pre-publication structural checks.

Preservation and portability

All 50 frozen snapshot JSONs and copied trial files are byte-identical to their accepted sources. Only the new top-level review JSONs were adapted (unique UI index, reference-image paths, pending review state and source attribution). Original sources remain untouched. Historical absolute paths inside logs, configs and provenance are archival, not website image dependencies.

The standard unified/DTC mesh assets are external dependencies, just as for benchmark600. The two extra Allsides folders used here are additionally bundled in `extraassets/allsidesobjects/. The review website needs only the included rendered reference images; to replay on another machine, install those folders into owgrobot/assets/allsidesobjects/` without overwriting differing existing assets. Frozen badge PNGs and table textures are included. No new simulation replay was performed during consolidation; structural and content-hash validation was performed.

The excluded red-triangle count bank (campaign40/accepted/count_003) is deliberately not included. Rejected candidates and other campaign trials remain in the original hard-subset-work/ tree, not all duplicated here.