Datasetpublic

iciclab/Robot_physics_interaction_with_Human_linguistic

Dual-Camera Speech and Interaction Dataset This directory contains the processed output of 50 synchronized recording sessions. Each session combines first-person video, third-person video, human speech, 30 fps image sequences, and a text label derived from the read-aloud script. For the research context behind the dataset, see PROJECT_README.md. Directory structure ACTION_NUMBER/ ├── first_person_view.mp4 ├── third_person_view.mp4 ├── audio.wav ├── text.txt ├──… See the full description on the dataset page: https://huggingface.co/datasets/iciclab/Robot_physics_interaction_with_Human_linguistic.

sourceHugging Faceupdated 4d agoView on Hugging Face

1likes369downloads

Dataset Card

Dual-Camera Speech and Interaction Dataset

This directory contains the processed output of 50 synchronized recording sessions. Each session combines first-person video, third-person video, human speech, 30 fps image sequences, and a text label derived from the read-aloud script.

For the research context behind the dataset, see PROJECT_README.md.

Directory structure

text

ACTION_NUMBER/
├── first_person_view.mp4
├── third_person_view.mp4
├── audio.wav
├── text.txt
├── label.json
├── first_person_view_camera/
│   ├── first_person_view_camera_1.jpg
│   ├── first_person_view_camera_2.jpg
│   └── ...
└── third_person_view_camera/
    ├── third_person_view_camera_1.jpg
    ├── third_person_view_camera_2.jpg
    └── ...

Files in each session

File or directory	Description
`first_person_view.mp4`	First-person video, 640 x 480, approximately 10 fps; the source image was rotated 180 degrees during recording
`third_person_view.mp4`	Third-person video, 424 x 240, approximately 5 fps
`audio.wav`	Corresponding two-channel human speech recording at 48 kHz
`first_person_view_camera/*.jpg`	First-person video converted to a 30 fps JPEG sequence
`third_person_view_camera/*.jpg`	Third-person video converted to a 30 fps JPEG sequence
`text.txt`	Clean script text associated with the session
`label.json`	Script identity, marked and clean text, ASR output, match scores, and review status

Frame numbering starts at 1. For either camera, frame N corresponds to (N - 1) / 30 seconds from the beginning of that camera stream.

Session naming

Processed session directories are numbered sequentially within each physical action:

text

press_01
press_02
pull_01
slide_01
twist_01
insert_01

The first component is the physical action, and the integer is the recording's chronological position within that action. Numbering starts at 1 for each action. Prosody and script information are retained in scenario_id and prosody_profile inside the label files rather than encoded in the directory name. The original timestamp directory name remains available as source_session.

Frame-rate conversion

The original first-person and third-person videos were recorded at approximately 10 fps and 5 fps, respectively. The image sequences were generated with the FFmpeg fps=30 filter. This operation repeats the nearest source frame on a 30 Hz timeline; it does not perform optical-flow or generative interpolation. Adjacent JPEG files may therefore contain the same source image.

The two cameras can have slightly different start and end delays. As a result, the first-person and third-person image counts within one session may differ. Each stream has its own frame index and timestamp beginning at zero.

Frame manifest

frame_manifest.csv contains one row per image and can be used directly to construct image-text samples.

Field	Description
`session`	Action-based processed directory name, such as `press_01`
`camera`	`first_person_view_camera` or `third_person_view_camera`
`frame_index`	One-based frame number
`timestamp_s`	Time from the beginning of the camera stream in seconds
`image_path`	Image path relative to this directory
`scenario_id`	Script identifier, such as `PRES-U01`
`text`	Clean script text associated with the session

Every image from the same session shares that session's utterance-level text label.

Session labels

label.json, session_labels.json, and session_labels.csv use the following main fields:

Field	Description
`source_session`	Original timestamp directory name, such as `session_20260918_120529`
`session`	Action-based processed directory name
`scenario_id`	Script identifier
`title`	Script-item title
`task`	Interaction type: `PRES`, `PULL`, `SLID`, `TWST`, or `INSR`
`prosody_profile`	Prosody class: `U`, `E`, `C`, or `R`
`marked_text`	Original script with performance directions, stress, and pause notation
`text`	Clean text used as the image label
`asr_transcript`	Whisper transcription of the recorded speech
`match_score`	Similarity between the ASR transcript and the selected script, from 0 to 1
`match_margin`	Difference between the best and second-best automatic match scores
`match_method`	`asr_fuzzy_match` or `sequence_override`
`needs_review`	`true` when the recording should be checked manually