mumbumble/GAZE_soundspace

Unity + SoundSpaces 2.0 Spatial Audio Dataset (Replica)

Audio-visual spatial-audio dataset built by placing sound-emitting objects in 17 Replica indoor scenes with a Unity/PhysX physical placement stage, then rendering binaural room impulse responses (RIR) with SoundSpaces 2.0 (habitat-sim AudioSensor, per-class acoustic materials). Each case ships the listener's first-person render, per-source binaural RIRs, dry source audio, the RIR-convolved per-source audio, a mixed binaural track, and full geometric metadata (2D/3D bounding boxes, listener-local source directions).

Scenes (17)

office_0..4, room_0..2, hotel_0 (small), frl_apartment_0..5, apartment_0..1 (large).

apartment_2 is excluded from the QA100 release because the Unity/PhysX placement stage was unstable for that scene on the current generation setup.

Augmentations (per scene)

Directory layout

<scene>/<augmentation>/case_<id>/
├── soundspaces/
│   ├── coords.json                      # full metadata (schema below)
│   ├── unity_fpv.png                    # listener first-person render (1280x960)
│   ├── soundspaces_validation_spheres.png
│   ├── unity_fpv_vs_soundspaces_spheres.png
│   ├── rir_src01_*.npy                  # binaural RIR, shape (2, N), float64
│   ├── audio_src01_*.wav                # dry source convolved with its RIR (16 kHz, stereo)
│   └── audio_mix.wav                    # all sources mixed (16 kHz, stereo)
└── unity/
    └── unity_coords.json                # raw Unity-side placement output

Audio format

• —Sample rate: 16 kHz, stereo (binaural), float.
• —RIR: NumPy (2, N) float64. Channel 0 = LEFT ear, channel 1 = RIGHT ear. Per-source audio = fftconvolve(dry_signal, rir[channel]).

coords.json schema — coordinate systems & directionality

Each case stores geometry in two coordinate frames:

Conversion: habitat = (x, y, -z). Listener facing: soundspaces_yaw = (90 + unity_yaw) % 360 (the +unityyaw accounts for the yaw direction reversed by the Z-flip; using `90 - unityyaw` mirrors the X axis and swaps the binaural L/R channels — that bug is fixed in this release).

Top-level keys:

• —scene, unity_yaw_deg, soundspaces_yaw_deg, image_width, image_height, camera_intrinsic [fx, fy, cx, cy]
• —listener_unity_world / listener_habitat [x, y, z], ear_height_offset_m
• —acoustic_materials (per-class material mapping report; 101 Replica classes)
• —sources: list, each with:
• —asset, class_num, audio_path, surface_mode, source_origin
• —unity_world_xyz / habitat_xyz — source position in each frame
• —`listener_local_xyz` — source in listener-facing local frame, the key directional field:
• —x < 0 = LEFT, x > 0 = right
• —z > 0 = FRONT, z < 0 = behind
• —y = up/down
• —bbox_2d [x0, y0, x1, y1] pixels (xyxy_top_left_pixels)
• —bbox_3d_unity_world / bbox_3d_habitat — axis-aligned [min_xyz, max_xyz]
• —scale_multiplier, object_yaw_deg, rotation_euler_deg, color_rgb
• —rir_file, wav_file, rir_shape

Binaural directionality note: for sources clearly off-axis (≳15°), the direct-path arrival lands in the correct ear (listener_local_xyz.x < 0 → louder/earlier LEFT). For near-frontal sources (≲10°) the direct-path level difference is tiny and early room reflections in asymmetric rooms can dominate the per-channel energy — this is expected binaural physics, not a labeling error.

Licensing & attribution — READ BEFORE PUBLIC RELEASE

This dataset is derived from the Replica Dataset (meshes, renders, semantic classes). Replica is distributed under its own research license, which restricts redistribution. Rendered images and semantic-derived fields here inherit those terms. Confirm you are permitted to redistribute Replica-derived content before making this dataset public.

• —Replica: https://github.com/facebookresearch/Replica-Dataset (see LICENSE)
• —SoundSpaces 2.0: https://github.com/facebookresearch/sound-spaces
• —Placed-object audio assets (data/Effect_Object/...): verify their individual licenses are compatible with redistribution.

Citation

If you use this dataset, please cite Replica and SoundSpaces 2.0.