nusshao/SocialNav-Dataset

SocialNav-SUB数据集制作

Background

Raw Dataset地址为 https://huggingface.co/datasets/michaelmunje/SocialNav-SUB

根据原论文，是一个VQA（Video Question Answering）的数据集。

但是作者只提供了Raw Data，没法直接用来训练VLM。此项目为制作openai-chat格式的VQA数据集，用于llama-factory的SFT训练。

前视图提取

作者提供的human-labeled的RGB图像，是front-view和BEV拼接在一起的。

<img alt="rawrgb.png" height="400" src="demoimages/raw_rgb.png" width="500"/>

我们运行extractlabeled.py将labeled RGB提取到/labeledfront_view文件夹中，同时保留num of people的信息。

<img alt="extractedrgb.png" height="200" src="demoimages/extracted_rgb.png"/>

最后，运行downsample.py将图片分辨率降采样为640*360。

问答对构建

运行build_dataset.py即可

问题类别

作者在论文中给出了对应label的多种问题：

Spatial Reasoning

• —qpersonspatialpositionbegin_p{id}: The position of person-{id} at the beginning of the video is
• —qpersonspatialpositionend_p{id}: The position of person-{id} at the end of the video is
• —qgoalposition_begin: The initial position of the goal with respect to the robot’s view is
• —qgoalposition_end: The end position of the goal with respect to the robot’s view is
• —qobstructingendpositionp{id}: Whether person-{id} is obstructing the robot’s path towards the goal at the end of the video?

Spatiotemporal Reasoning

• —qrobotmoving_direction: The direction of the robot moving in the video is
• —qpersondistancechangep{id}: The relative distance change of person-{id} to the robot from the beginning of the video to the end is
• —qobstructingpath_p{id}: Whether person-{id} is obstructing the robot’s path towards the goal during the video?

Social Reasoning

• —qrobotaffected_p{id}: Whether the robot’s actions are affected by person-{id}?
• —qrobotaction_p{id}: The high-level relational action of the robot with respect to person-{id} is
• —qpersonaffected_p{id}: Whether person-{id}’s actions are affected by the robot?
• —qpersonaction_p{id}: The high-level relational action of person-{id} with respect to the robot is
• —qrobotsuggestedaffectedp{id}: Whether the robot’s actions are affected by person-{id} at the end of the video?
• —qrobotsuggestedactionp{id}: The high-level relational action of the robot with respect to person-{id} at the end of the video is
• —qhumanfutureactionprediction_p{id}: The high-level relational action of person-{id} with respect to the robot at the end of the video is

Label获取

这里为了构建label方便，我们直接取possibility最高的一个答案作为label。 例如

"qobstructingendpositionp2": [ "no", "yes" ],

"qobstructingendpositionp2_probabilities": [ 0.9, 0.1 ]

那么我们的label=“no”

openai-chat格式的问答对构建

根据我们定义的问题以及获取的label，可以构建openai-chat格式的问答对。如下：

"conversations": [ { "from": "human", "value": "<image>xN Based on the provided video, answer: Whether person-{id} is obstructing the robot’s path towards the goal at the end of the video?" }, { "from": "gpt", "value": "no" } ],

添加图像源

最终，加上N张图像的路径：

{ "conversations": [ { "from": "human", "value": "<image>xN Based on the provided video, answer: Whether person-{id} is obstructing the robot’s path towards the goal at the end of the video?" }, { "from": "gpt", "value": "no" } ], "images": [ "labeledfrontview/{subfoldername}/1.png", "labeledfrontview/{subfoldername}/2.png", ... ] },

配置dataset info

{ "socialnav": { "filename": "dataset.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "images": "images" }, "tags": { "roletag": "from", "contenttag": "value", "usertag": "human", "assistant_tag": "gpt" } } }