behavior-in-the-wild/BLIFT

BLIFT: Behavior-LLaVA Instruction Fine-Tuning Dataset

Paper: **Teaching Human Behavior Improves Content Understanding Abilities of VLMs**

Website: https://behavior-in-the-wild.github.io/behavior-llava.html

Dataset Summary

BLIFT (Behavior-LLaVA Instruction Fine-Tuning) is a large-scale multimodal instruction tuning dataset designed to teach Vision-Language Models (VLMs) human behavior. It contains over 730k images and videos collected from Reddit and YouTube, annotated with reciever behavior such as comments, likes, views, and replay graphs.

By modeling these downstream receiver behaviors, training on BLIFT improves content understanding of VLMs, showing significant improvements across 46 tasks in image, video, text, and audio understanding.

<img src="./bllava-fig_2.png" alt="bllava-fig" width="1000"/>

Dataset Structure

Each sample in BLIFT includes:

Data Sources

BLIFT combines high-quality behavioral data from two sources:

Reddit

• —Subreddits: r/pics, r/videos
• —Collected: 400k images, 330k videos
• —Metadata: Upvotes and top comments
• —Filtering: NSFW, bots, duplicates, minimum comment quality

YouTube

• —250k videos from ~6,000 verified channels via Wikidata
• —Metadata: Likes, views, top comments, replay graphs
• —Filtering: English language, minimum 10k views, NSFW, duplicates

<img src="./filtering-final.png" alt="filtering" width="1000"/>

Benchmarks & Results

Using BLIFT to train Behavior-LLaVA (a fine-tuned LLaMA-Vid), the model outperforms base LLaMA-Vid and other supervised baselines on:

• —46 tasks
• —26 benchmark datasets
• —Across image, video, audio, and text modalities

<img src="./radar_chart (1).png" alt="results" width="1000"/>

🔗 Citation

If you use BLIFT, please cite:

@article{singh2024teaching,
            title={Teaching Human Behavior Improves Content Understanding Abilities Of LLMs},
            author={Singh, Somesh and SI, Harini and Singla, Yaman K and Baths, Veeky and Shah, Rajiv Ratn and Chen, Changyou and Krishnamurthy, Balaji},
            journal={arXiv preprint arXiv:2405.00942},
            year={2024}
          }

Contact

Contact behavior-in-the-wild@googlegroups.com for questions and suggestions.