OakVolcano7858/ImmersiveMC-data

MCGestureControl Crowdsourced Data

Training data for the MCGestureControl project — an AI system that plays Minecraft using body gestures captured by a phone camera.

Dataset Structure

minecraft/    — Gameplay telemetry NPZ files (game_state, controls, timestamps)
body/         — Body skeleton NPY files (260-dim: pose + hands per frame)
paired/       — Paired gameplay + body recordings

Data Formats

Minecraft Gameplay (NPZ)

• —controls — (T, 28) float32: resolved game controls per frame
• —game_state — (T, 46) float32: game state vectors per frame
• —timestamps — (T,) float64: unix timestamps
• —frame_indices — (T,) int32: corresponding video frame indices
• —fps — float32: recording framerate (typically 30)
• —version — int32: recorder version
• —recorder_id — string: anonymous machine identifier

Body Skeleton (NPY)

• —Shape: (T, 260) float32
• —Layout: poseworld(99) + lefthand3d(63) + righthand3d(63) + leftpresent(1) + rightpresent(1) + posevisibility(33)

Controls Layout (28 dims)

Movement (0-6), Attack/Use (7-8), Look yaw/pitch (9-10), Utility (11-13), Hotbar one-hot (14-22), Cursor (23-24), Inventory clicks (25-27)

Contributing

Data is collected via the MCGestureControl crowdsourcing portal. Contributors register, accept a data agreement, and upload recordings.

Privacy

• —Raw body video is auto-processed into skeleton data and deleted
• —Skeleton data contains no personally identifiable information
• —Contributor IPs are stored as irreversible SHA-256 hashes