yulingxi/Metric-Bench

Metric-Bench Test

Metric-Bench Test set evaluates metric spatial understanding from indoor RGB images and explicit anchor measurements. Each example asks for a physical measurement of a referred object or the distance between two objects. Inputs consist of an image and an English question; the reference answer is a numeric JSON object.

This release contains 1,340 questions, 134 images and 20 scenes, with one test split. It is a reconstructed test revision and does not reproduce or re-evaluate previously reported paper scores. No training or validation split is included.

Dataset composition

Each image has ten questions. Questions provide either two anchors (539 examples) or three anchors (801 examples).

Repository structure

README.md
test.parquet
dataset_stats.json
SHA256SUMS
images/
  scene0084_00/
    1130.jpg
  ...

test.parquet is the canonical evaluation table. Image paths are relative to the repository root. Images are stored separately as JPEG files; the table contains path strings, not embedded image bytes or automatically decoded Hugging Face Image features. dataset_stats.json contains aggregate counts and packaging checks. SHA256SUMS covers all other release files.

Data fields

Ground truth uses a JSON string so scalar and vector answers share one stable column type. For example, {"length":0.388} and {"center":[0.037,-0.633,2.712]} are valid reference answers. Numeric precision is preserved from the source export.

Loading

Install datasets, huggingface_hub and Pillow. From a downloaded repository, run:

import json
from pathlib import Path
from datasets import load_dataset
from PIL import Image

root = Path(".")  # Repository root
test = load_dataset(
    "parquet", data_files={"test": str(root / "test.parquet")}, split="test"
)
example = test[0]
with Image.open(root / example["images"][0]) as image:
    image = image.convert("RGB")
question = example["question"]
reference = json.loads(example["ground_truth"])

After publication, download the table and resolve each image from the same repository revision:

import json
from datasets import load_dataset
from huggingface_hub import HfApi, hf_hub_download
from PIL import Image

repo_id = input("Hugging Face dataset repository ID: ").strip()
revision = HfApi().dataset_info(repo_id).sha
test = load_dataset(repo_id, revision=revision, split="test")
example = test[0]
image_path = hf_hub_download(
    repo_id=repo_id, repo_type="dataset", revision=revision,
    filename=example["images"][0],
)
with Image.open(image_path) as image:
    image = image.convert("RGB")
reference = json.loads(example["ground_truth"])

The explicit split configuration follows the Hugging Face dataset configuration documentation. Loading the table alone does not download the separate JPEG files.

Measurement conventions

All measurements are in meters.

• —width, height and length are physical object size parameters; dimension orders them as [width, height, length].
• —center is [x, y, z] in camera coordinates. depth is its camera-Z component.
• —box concatenates camera center and physical dimensions as [x, y, z, width, height, length]. The size parameters are not camera-axis-aligned bounding-box extents, and this representation does not include object rotation.
• —elevation is the object's bottom elevation in the source world coordinate system: world_center_z - world_height / 2. It is not necessarily height above the local floor.
• —distance is the nearest-surface distance between the two source world-axis-aligned bounding boxes, not the distance between object centers or reconstructed meshes.

Evaluation use

Provide only the image and question as model input. Ask for one JSON object with the key specified by question_type and a numeric scalar or array in the order above. For example: Return only a JSON object with key "length" and a numeric value in meters. No explanation or intermediate reasoning is required. Keep ground_truth outside the model input and use it only for scoring.

This release provides reference answers but does not ship an official scoring implementation or specify a paper-equivalent acceptance threshold. For reproducible evaluations, report the repository revision, model version, prompt, decoding settings, invalid-output handling and numeric tolerance or error metric. Report results by question type; scalar and vector outputs require explicit scoring definitions. State whether aggregate results are weighted by example count or averaged across question types, because width has twice as many examples as each other type.

The intended use is held-out evaluation of visual metric estimation with supplied anchors. Training or tuning on the released answers compromises this use. Multiple questions share each image and scene, so question-level random partitions are not independent scene splits.

Source and preparation

RGB images and underlying scene annotations originate from ScanNet. This release preserves the source export's question text, reference answers, identifiers and JPEG bytes. Packaging removes internal curation metadata, machine-specific paths, auxiliary target geometry and conversation instructions requesting intermediate reasoning.

Source curation included geometry checks and automated visual/text review; it does not establish human verification of every example. The release packaging checks cover record uniqueness, answer preservation, Parquet round-trip integrity, relative image paths, image decoding and source image hashes. They do not constitute a fresh independent geometric or semantic audit.

Limitations

The benchmark covers a small set of indoor scenes and supplied anchor measurements. Performance should not be generalized to outdoor scenes, unseen domains or unassisted metric estimation. Referring expressions, occlusion and source geometry may introduce ambiguity or annotation errors. Axis-aligned source boxes approximate object surfaces, which affects the interpretation of distance. This package alone does not establish disjointness from any external training corpus.

Data terms and attribution

ScanNet data is governed by the ScanNet Terms of Use, as stated in the official ScanNet repository. This packaging does not grant additional rights to the source images or annotations. Public image redistribution must be covered by the publisher's applicable permissions. No separate license for the benchmark-authored questions is supplied in the source package; this card does not assign one on the authors' behalf.

When reporting results, identify this release as Metric-Bench Test and include its repository revision. Credit the underlying ScanNet dataset using the citation provided in the official ScanNet repository. No benchmark paper identifier or author list is supplied with this package, so a benchmark bibliographic citation is not fabricated here.