THULab/ego4d_manipulation_v1

ego4dmanipulationv1 (TsFile)

Apache TsFile version of `mderry/ego4d-manipulation-v1`.

• —Records: 25,415

Schema (TsFile structure)

• —episode_index, task_index (TAG) — device dimension(s).
• —episode_index (FIELD).
• —task_index (FIELD).
• —frame_index (FIELD).
• —observation_images_ego (FIELD).
• —observation_state_0 (FIELD).
• —observation_state_1 (FIELD).
• —observation_state_2 (FIELD).
• —observation_state_3 (FIELD).
• —observation_state_4 (FIELD).
• —observation_state_5 (FIELD).
• —observation_state_6 (FIELD).
• —observation_state_7 (FIELD).
• —observation_state_8 (FIELD).
• —observation_state_9 (FIELD).
• —observation_state_10 (FIELD).
• —observation_state_11 (FIELD).
• —observation_state_12 (FIELD).
• —observation_state_13 (FIELD).
• —action_0 (FIELD).
• —action_1 (FIELD).
• —action_2 (FIELD).
• —action_3 (FIELD).
• —action_4 (FIELD).
• —action_5 (FIELD).
• —action_6 (FIELD).
• —action_7 (FIELD).
• —action_8 (FIELD).
• —action_9 (FIELD).
• —action_10 (FIELD).
• —action_11 (FIELD).
• —action_12 (FIELD).
• —action_13 (FIELD).

Usage

Install the Apache TsFile Python SDK (pip install tsfile) and read a converted file:

from pathlib import Path
from tsfile import TsFileReader

path = Path("data/ego4d_manipulation_v1.tsfile")
with TsFileReader(str(path)) as reader:
    schemas = reader.get_all_table_schemas()
    print("tables:", list(schemas))
    table_name = next(iter(schemas))
    table = schemas[table_name]
    columns = [column.get_column_name() for column in table.get_columns()]
    print("columns:", columns)
    field_names = [
        column.get_column_name()
        for column in table.get_columns()
        if column.get_column_name() not in {"Time", "time"}
    ]
    if field_names:
        with reader.query_table(table_name, field_names[:3], batch_size=1024) as result:
            batch = result.read_arrow_batch()
            if batch is not None:
                print(batch.to_pandas().head())

Source & license

• —Original dataset: https://huggingface.co/datasets/mderry/ego4d-manipulation-v1
• —Author / publisher: mderry
• —License: not declared by the original dataset; please defer to the original.