zeroshotdata/sf_fold

<div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/light_logo.svg" width="50%" alt="Zeroshot Data" /> </div>

<div align="center"> <a href="https://www.zeroshotdata.com/" target="blank"> <img alt="Homepage" src="https://storage.googleapis.com/zeroshot-public-rrds/assets/zeroshotbadge.svg" /> </a> <a href="https://x.com/ZeroShotData" target="blank"> <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-ZeroshotData-white?logo=x&logoColor=white" /> </a> <a href="https://huggingface.co/datasets/zeroshotdata/zeroshot-test-dataset" target="_blank"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Zeroshot%20Data-ffc107?color=ffc107&logoColor=white" /> </a> </div> <h2> <p style="color: #594EE6;" align="center"> <span style="color: #594EE6;">SF Fold</span> </p> </h2> <h3 align="center"> <span style="color: #594EE6;">Real-world data for general robotics.</span> </h3>

<br/> <br/>

🤖 What is SF Fold?

<p>This dataset contains real-world residential t-shirt folding demonstrations, collected by trained data collectors using hand-held grippers in diverse home environments.</p>

📖 Table of Contents

1. 1.Features
2. 2.Terminology
3. 3.Specifications
4. 4.Dataset Composition
5. 5.Environment Composition
6. 6.Garment Composition
7. 7.Trajectory Specifications
8. 8.Hardware Specifications
9. 9.Physical Specifications
10. 10.Data Format
11. 11.Conventions
12. 12.Directory Layout
13. 13.Video Files
14. 14.Parquet Format
15. 15.Metadata Files
16. 16.Frame of Reference
17. 17.Collection
18. 18.Collection Info
19. 19.Privacy & Consent
20. 20.T-Shirt Folding Procedure Guidelines
21. 21.T-Shirt Folding Instructions
22. 22.Preprocessing & Annotation
23. 23.Validation
24. 24.Distribution
25. 25.Appendix

📊 Features

This dataset contains real-world residential t-shirt folding demonstrations, collected using hand-held grippers in diverse household environments with varied lighting differences.

📖 Terminology

🛠️ Specifications

Dataset Composition

Environment Composition

<div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/environment_stats.png" width="70%" alt="Environment Composition" /> </div>

Garment Composition

Garments include short-sleeve T-shirts of varying materials (cotton, polyester, scrubs) and collars (crew, polo).

Trajectory Specifications

Hardware Specifications

Camera

Gripper Encoder

Physical Specifications

Frame Reference

<div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/3d_puppet.png" width="70%" alt="3D Puppet Coordinate Frames Visualization" /> </div>

Frame Values

• —Tpuppetimu (origin)

  [[ 1.        0.        0.        0.01625 ]
    [ 0.        1.        0.       -0.016424]
    [ 0.        0.        1.        0.189372]
    [ 0.        0.        0.        1.      ]]

• —Tjointaxis (origin to end actuator mounting location)

  [[ 1.        0.        0.        0.01685 ]
    [ 0.        1.        0.       -0.032924]
    [ 0.        0.        1.       -0.03476 ]
    [ 0.        0.        0.        1.      ]]

• —Tnormalfingertip (origin to end actuator fingertip)

  [[ 1.        0.        0.        0.01635 ]
    [ 0.        1.        0.       -0.016538]
    [ 0.        0.        1.        0.134907]
    [ 0.        0.        0.        1.      ]]

• —Tpuppetcamera (origin to camera)

  [[1.       0.       0.       0.0163  ]
  [0.       1.       0.       0.035513]
  [0.       0.       1.       0.02112 ]
  [0.       0.       0.       1.      ]]

🗂️ Data Format

Dataset follows the [LeRobot Standard](https://docs.phospho.ai/learn/lerobot-dataset) for robotic learning.

Conventions

• —Episodes: atomic trajectories (episode_index)
• —Frames: time-ordered (frame_index)
• —Timestamps: monotonic
• —Observations: camera-aligned signals (images, proprioception).
• —Actions: commanded end-effector pose + gripper width.
• —State: measured end-effector pose + gripper width.
• —Coordinate frames: world (static), camera (optical center), motion_axis, fingertip.
• —Units: meters, seconds, radians
• —Rotations: quaternions (qx, qy, qz, qw) with right-handed, w last.

Directory Layout

    sf-fold/
    ├── meta/
    │ ├── info.json                         # global metadata
    │ ├── tasks.jsonl                       # per-task descriptors
    │ ├── episodes.jsonl                    # per-episode descriptors
    │ ├── episodes_stats.jsonl              # per-episode statistics
    │ ├── masks.jsonl                       # coordinates to mask each episode
    │ └── camera_intrinsics.jsonl           # camera properties for each episode
    ├── data/
    │   └── chunk-{000..}/episode_{000000..}.parquet
    └── videos/
        └── chunk-{000..}/
          └── observation.images.cam_ego/episode_{000000..}.mp4
          └── observation.images.cam_left/episode_{000000..}.mp4
          └── observation.images.cam_right/episode_{000000..}.mp4

Video Files

Three synchronized camera streams per episode:

• —cam_ego: First-person view
• —cam_left: Left puppet view
• —cam_right: Right puppet view

Accessed as VideoFrame objects via Hugging Face interfaces.

Parquet Format

Each row includes:

Example Parquet Data

"index": 124578,
"frame_index": 318,
"timestamp": 12.634,
"episode_index": 42,
"gripper_width": [0.034, 0.034],
"task_index": 7,
"left_camera_pose": [0.152, -0.031, 0.884, 0.002, 0.713, -0.001, 0.701],
"right_camera_pose": [0.148, 0.029, 0.882, -0.003, 0.710, 0.006, 0.704],
"left_fingertip_pose": [0.612, -0.084, 0.502, 0.002, 0.005, 0.721, 0.693],
"right_fingertip_pose": [0.616, 0.089, 0.503, -0.003, 0.004, 0.718, 0.696]

Metadata Files

• —info.json — Global dataset metadata, including:

• —codebase_version (e.g., v2.1)
• —robot_type (e.g., zeroshot)
• —Episode and frame counts (total_episodes, total_frames)
• —Chunking details (total_chunks, chunk_size)
• —Data and video path templates
• —Global features schema (per-signal dtype, shape, pose keys, video codec info, etc.)

• —tasks.jsonl — Line-delimited JSON defining tasks:

• —task_id
• —name

• —episodes.jsonl — Line-delimited JSON describing episodes:

• —episode_id / episode_index
• —Associated task_id

• —episodes_stats.jsonl — Line-delimited JSON with per-episode statistics:
• —episode_id / episode_index
• —num_frames
• —duration (seconds)
• —Error flags (e.g., missing frames, dropped sensors)

Frame of Reference

All trajectories and poses are expressed in a static world frame. The world frame origin is defined at the base of the right puppet, serving as the global reference for all coordinate transforms. Camera poses, gripper positions, and fingertip poses are aligned to this frame, with translations given in meters and orientations expressed as quaternions (qx, qy, qz, qw).

🛒 Collection

This dataset was primarily collected in San Francisco, California, USA. Data collection took place in noisy and diverse real-world environments to capture a broad range of variability. All sessions were performed by trained, paid data collectors following standardized procedures to ensure consistency across sessions.

To enhance robustness, failure cases—such as tangled fabrics or irregular interactions—were intentionally retained, increasing the diversity of captured scenarios. Each recording was reviewed for quality, and only those meeting data standards were included in the final dataset.

Collection Info

• —Location: San Francisco, California, USA 🇺🇸
• —Collectors: Trained, paid data collectors
• —Diversity: Includes both successful and failure cases

Privacy & Consent

All data collection was conducted under protocols designed to ensure privacy and informed consent. Participants were explicitly notified of the scope of collection, including video, image, IMU, and encoder streams, and provided documented consent prior to recording. Permission to capture data was obtained for all environments.

T-Shirt Folding Procedure Guidelines

• —Fixed stance: Torso facing workspace
• —Hardware: Only in-house system used
• —Variability: Natural irregularities retained (no retries)

The t-shirt folding task was performed under standardized collection protocols to ensure reproducibility and consistency across sessions. Data collectors maintained a fixed stance throughout demonstrations, with feet planted and torso oriented toward the workspace, avoiding lateral rotation or excessive forward lean.

All manipulations were conducted exclusively using the in-house hardware system. When irregularities occurred—such as tangled fabric or misaligned folds—the procedure continued rather than being restarted, preserving the natural variability of the task.

Across all episodes, data collectors executed the folding sequence with the goal of efficient task completion, minimizing extraneous motion while maintaining data fidelity.

T-Shirt Folding Instructions

1. 1.Retrieve garment a. Pick up a shirt from the laundry basket.

1. 1.Position shirt a. Lay the shirt flat on a surface, front side down, with the collar aligned at the top.

1. 1.Fold sides a. Fold one side inward to the shirt’s centerline. b. Fold the remaining side inward so both edges overlap neatly. <div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/fold_0.jpeg" width="70%" alt="Folding Procedure Image 0" /> </div>

1. 1.Create folds a. Fold the shirt upward from the bottom to the midline. <div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/fold_1.jpeg" width="70%" alt="Folding Procedure Image 1" /> </div>

b. Fold again from the midline to the collar to form a compact rectangle. <div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/fold_2.jpeg" width="70%" alt="Folding Procedure Image 2" /> </div>

1. 1.Stack a. Place the folded shirt neatly onto the prepared stack or storage area. <div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/fold_3.jpeg" width="70%" alt="Folding Procedure Image 3" /> </div>

✒️ Preprocessing & Annotation

Preprocessing was performed using a lightweight filtering pipeline designed to stabilize signals while preserving fine motion. A Fixed-Interval Kalman filter with a Rauch–Tung–Striebel (RTS) smoother was applied to refine trajectory estimates.

Rotational data and gripper width signals were further processed using a double Butterworth filter to reduce noise while maintaining sharp transitions. Filtering was implemented with standard numerical libraries within an in-house software framework.

The filtering strategy was kept conservative, ensuring that subtle manipulations and fine-scale movements remained intact. All raw, unfiltered recordings were preserved and remain available upon request.

Preprocessing

Applied lightweight filtering with: | Filter / Method | Applied To | |------------------|-------------| | Kalman + RTS Smoother | Trajectories | | Double Butterworth | Rotational data, gripper width | | Light Filtering | All signals | | Raw Data Retention | All modalities |

Labeling

• —Room-type labels (bedroom, kitchen, etc.)
• —Contextual comments and anomalies
• —Optional fine-grained sub-task labels available

🔬 Validation

Validation was performed by recording a range of manipulation motions using the in-house hardware system, while simultaneously capturing ground truth with an **OptiTrack Trio 32** motion capture system. Signals were time-aligned, and frame-wise errors were computed for both position and orientation measurements.

3D Position Accuracy

Free-space trajectories were executed, including sweeping and randomized motions of the end-effector. Errors were computed frame-by-frame against OptiTrack ground truth data. The mean 3D positional error was consistently within the low-centimeter range, with additional improvements observed after applying the Fixed-Interval Kalman filter with RTS smoothing. These results confirm the reliability and consistency of positional tracking across diverse motion patterns.

Orientation Accuracy

Orientation estimates were validated against quaternion data from the OptiTrack system. A double Butterworth filter was applied to the raw rotational signals, effectively reducing jitter while preserving sharp transitions. Across all validation trials, the system achieved sub-degree median orientation errors, demonstrating fine angular precision suitable for high-accuracy manipulation tasks.

🚚 Distribution

ZeroShot is responsible for maintaining this dataset. For any questions or concerns, please contact [interest@zeroshotdata.com](mailto:interest@zeroshotdata.com).

The dataset will be released as an open-source resource under the Apache 2.0 license and made publicly available via a public Google Cloud Storage Bucket. A planned public release is scheduled, with subsequent updates distributed through the same platform. Versioning will allow users to reference specific releases. All updates will be documented in this file and reflected across relevant distribution points. Older versions may be archived for reproducibility, and any obsolescence will be clearly communicated.

Licensing and terms of use are currently under review (expected to align with a modified BSD 3-Clause license where redistribution may require permission). Any restrictions on commercial use, redistribution, or derivative works will be specified once finalized.

The dataset contains no third-party rights or export-control restrictions. Access will be free of charge, and users are encouraged to provide proper attribution by citing ZeroShot and the associated technical documentation when referencing this dataset in publications.

🔗 Appendix

Data Samples

<div align="center"> <img src="https://storage.googleapis.com/zeroshot-public-rrds/assets/rerun_preview.jpeg" width="100%" alt="Dataset Sample Image" /> </div>

Interactive Samples

📚 Citation

@dataset{zeroshot_tshirt_2025,
  title     = {ZeroShot T-shirt Folding Dataset},
  author    = {ZeroShot Data Team},
  year      = {2025},
  note      = {Version 1.2, September 2025},
  url       = {https://zeroshotdata.com/datasets/tshirt}
}