Wojtekb30/HumanML3D-500ms-FPP-descriptions-CoTs-1

HumanML3D 500ms First person perspective descriptions for CoTs

Introduction

This repository contains files of the Mr. Ri's and Ms. Tique's HumanML3D human motion dataset, <br>but also descriptions of the movements in first person perspective in 0.5 second time windows.

The descriptions were created synthetically with use of a multimodal LLM and are in json format. They can be found in ``comics_and_descriptions`` folder.

The dataset also contains motion capture data and files used in process of generating the detailed descriptions.

Link to the original dataset this one derives from: https://www.kaggle.com/datasets/mrriandmstique/humanml3d

Intended use

Intended use of this dataset is research on movement generation in anthropomorphic robots and avatars.

The descriptions are in first person perspective, making them possible to use in LLM chains of thought.

Structure

Files related to specific movement recording have same filename, for example 000000.txt/json/npy.

The json files contain:

• —global_intent - general description of the move as a whole, not necessarily in 1st person.
• —globalintentme - same as above but in 1st person.
• —globalintentcmd - the move as a command.
• —motion_conclusion - a sentence in 1st person that concludes the whole move
• —steps - contains descriptions of what the character is doing in 1st person every 0.5 seconds.
• —step - number
• —time - time window (sometimes little less than 0.5 seconds)
• —thought - description of the time frame in 1st person

An full example is underneath, later in this readme.

Since HF has a file amount limit, I had to ZIP the folders. But you can view several file samples from the ZIPped directories in ``data_examples`` folder.

The pipeline

Motion capture files were first rendered into stick figure "comics" that show the movement, <br>and text files describing limb movements were created thru reading the info from the motion capture files, for example:

Original description: a man full-body sideways jumps to his left.

Step 1 (Time: 0.0s - 0.5s):
  Right hand: down, forward
  Left hand: right, down
  Head: right, down, forward
  Pelvis (Root): down, backward

Step 2 (Time: 0.5s - 0.9s):
  Right hand: right, up, backward
  Left hand: right, up, backward
  Right foot: right, up, backward
  Left foot: right, up, backward
  Head: right, up, backward
  Pelvis (Root): right, up

Step 3 (Time: 1.0s - 1.4s):
  Right hand: right, down, backward
  Left hand: right
  Right foot: right
  Left foot: right, down
  Head: right, down, backward
  Pelvis (Root): right, down, backward

Step 4 (Time: 1.5s - 1.9s):
  Right hand: up
  Left hand: left, down
  Head: right, backward

Step 5 (Time: 2.0s - 2.5s):
  Left hand: backward

Step 6 (Time: 2.5s - 3.0s):
  Left hand: right

Step 7 (Time: 3.0s - 3.5s):
  Right hand: backward

Step 8 (Time: 3.5s - 4.0s):
  [No changes to pose]

The comics and text files were then rewritten using Gemma3:4B multimodal LLM, resulting in files like:

{
    "global_intent": "I am lifting an object from the left and placing it on the right.",
    "global_intent_me": "I reach my left hand forward, then lift the object upwards.",
    "global_intent_cmd": "Lift left, place right.",
    "steps": [
        {
            "step": 1,
            "time": "0.0s - 0.5s",
            "thought": "I extend my left hand forward, ready to grasp."
        },
        {
            "step": 2,
            "time": "0.5s - 0.9s",
            "thought": "I begin to lift the object upwards slowly."
        },
        {
            "step": 3,
            "time": "1.0s - 1.4s",
            "thought": "I continue to lift the object steadily upwards."
        },
        {
            "step": 4,
            "time": "1.5s - 1.9s",
            "thought": "I hold the object high above my head now."
        },
        {
            "step": 5,
            "time": "2.0s - 2.5s",
            "thought": "I shift my weight to my right, preparing to move."
        },
        {
            "step": 6,
            "time": "2.5s - 3.0s",
            "thought": "I move my left hand to the right side slowly."
        },
        {
            "step": 7,
            "time": "3.0s - 3.5s",
            "thought": "I lower the object as I move it right."
        },
        {
            "step": 8,
            "time": "3.5s - 4.0s",
            "thought": "I continue moving my hand to the right side."
        }
    ],
    "motion_conclusion": "My hand is resting, palm up, near the edge of the counter."
}

Since I originally forgot about the motion_conclusion, I generated them later using Gemma2:2B.

Small models were used due to time constrains I had to create the descriptions.

All Python files are in the repository too with order written in their filenames as parts.<br> You can run the pipeline again if you wish, the scripts use Ollama so replacing the LLM is very easy. But due to LLMs not having deterministic nature, result descriptions will be different.

Thank you if you use this dataset, I am happy to hear what you created with it.