louisye/social-robotics-acoustic-prosody

Social Robotics: Acoustic Prosody (03c)

Ambient vocal tone around each task — alarming vs soothing — as corroborating context.

One layer of the Social-Affective Filter (SAF) — dehydrated social-signal metadata extracted from egocentric (first-person) video so robots can learn to read human reactions. No raw pixels and no audio. Each row is one source video, keyed by video_id; rehydrate against your own legally-obtained Ego4D copies (below).

• —Rows: 989 — videos in the evaluation slice for which this layer produced a measured signal (videos it could not measure are excluded from this per-layer dataset; the layers still join 1:1 on video_id).
• —Signal: A signed −1…+1 prosody scalar summarizing the acoustic tone of the task window (negative = alarming/discouraging, ~0 = neutral, positive = soothing/positive).
• —Method: emotion2vec+ / SenseVoice speech-emotion + librosa acoustic features on the task audio window.

⚠️ Read this first — interpretation caveats

• —Ambient audio, NOT bystander-attributed. Egocentric audio is dominated by the camera-wearer; there is no speaker separation. Use prosody_scalar only as corroboration for a per-bystander visual signal, never standalone.
• —Check audio_present in the raw column before fusing; tasks with no audio should be excluded, not read as confident-neutral.
• —Egocentric footage is legitimately low-yield (small/sparse bystander faces, heavy camera motion); we publish honest measurements only, never fabricated zeros.

Columns

Identity & manifest (shared across all SAF datasets)

Acoustic Prosody signal

The *_raw JSON column

The *_raw column holds the full nested per-task / per-person detail as a JSON string. Parse it with:

import json, pandas as pd
df = pd.read_parquet("hf://datasets/louisye/social-robotics-acoustic-prosody/social_metadata.parquet")
raw_col = next(c for c in df.columns if c.endswith("_raw"))
detail = json.loads(df.iloc[0][raw_col])

How to load

import pandas as pd
df = pd.read_parquet("hf://datasets/louisye/social-robotics-acoustic-prosody/social_metadata.parquet")
# or:  from datasets import load_dataset;  ds = load_dataset("louisye/social-robotics-acoustic-prosody")

Rehydration — mapping back to video

video_id is the Ego4D clip UUID. With your own licensed Ego4D copy, the file is <video_id>.mp4; timestamps in the *_raw columns index into that clip. A helper (rehydrate_dataset.py) is included. We never redistribute source media — obtain Ego4D under its own license.

Provenance

Generated by the SAF pipeline (export_metadata.json records schema_version + pipeline_git_sha). Headers are the descriptive layer name + metric; the pipeline's internal 03c_ layer-id prefix is stripped at publish time. License MIT (this metadata only; Ego4D videos remain under the Ego4D license).