hdong51/MultiOOD

<div align="center">

<h1>MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities</h1>

<div> <a href='https://sites.google.com/view/dong-hao/' target='blank'>Hao Dong</a>¹ &emsp; <a href='https://viterbi-web.usc.edu/~yzhao010/' target='blank'>Yue Zhao</a>² &emsp; <a href='https://chatzi.ibk.ethz.ch/about-us/people/prof-dr-eleni-chatzi.html' target='blank'>Eleni Chatzi</a>¹ &emsp; <a href='https://people.epfl.ch/olga.fink?lang=en' target='blank'>Olga Fink</a>³ </div> <div> ¹ETH Zurich, ²University of Southern California, ³EPFL </div>

<div> <h4 align="center"> • <a href="https://arxiv.org/abs/2405.17419" target='_blank'>arXiv</a> • </h4> </div>

<div style="text-align:center"> <img src="multiood.jpg" width="100%" height="100%"> </div>

</div>

MultiOOD is the first-of-its-kind benchmark for Multimodal OOD Detection, characterized by diverse dataset sizes and varying modality combinations.

Code

https://github.com/donghao51/MultiOOD

MultiOOD Benchmark

MultiOOD is based on five public action recognition datasets (HMDB51, UCF101, EPIC-Kitchens, HAC, and Kinetics-600).

Prepare Datasets

1. 1.Download HMDB51 video data from link and extract. Download HMDB51 optical flow data from link and extract. The directory structure should be modified to match:

HMDB51
├── video
|   ├── catch
|   |   ├── *.avi
|   ├── climb
|   |   ├── *.avi
|   |── ...


├── flow
|   ├── *_flow_x.mp4
|   ├── *_flow_y.mp4
|   ├── ...

1. 1.Download UCF101 video data from link and extract. Download UCF101 optical flow data from link and extract. The directory structure should be modified to match:

UCF101
├── video
|   ├── *.avi
|   |── ...


├── flow
|   ├── *_flow_x.mp4
|   ├── *_flow_y.mp4
|   ├── ...

1. 1.Download EPIC-Kitchens video and optical flow data by

bash utils/download_epic_script.sh 

Download audio data from link.

Unzip all files and the directory structure should be modified to match:

EPIC-KITCHENS
├── rgb
|   ├── train
|   |   ├── D3
|   |   |   ├── P22_01.wav
|   |   |   ├── P22_01
|   |   |   |     ├── frame_0000000000.jpg
|   |   |   |     ├── ...
|   |   |   ├── P22_02
|   |   |   ├── ...
|   ├── test
|   |   ├── D3


├── flow
|   ├── train
|   |   ├── D3
|   |   |   ├── P22_01
|   |   |   |     ├── frame_0000000000.jpg
|   |   |   |     ├── ...
|   |   |   ├── P22_02
|   |   |   ├── ...
|   ├── test
|   |   ├── D3

1. 1.Download HAC video, audio and optical flow data from link and extract. The directory structure should be modified to match:

HAC
├── human
|   ├── videos
|   |   ├── ...
|   ├── flow
|   |   ├── ...
|   ├── audio
|   |   ├── ...

├── animal
|   ├── videos
|   |   ├── ...
|   ├── flow
|   |   ├── ...
|   ├── audio
|   |   ├── ...

├── cartoon
|   ├── videos
|   |   ├── ...
|   ├── flow
|   |   ├── ...
|   ├── audio
|   |   ├── ...

1. 1.Download Kinetics-600 video data by

wget -i utils/filtered_k600_train_path.txt

Extract all files and get audio data from video data by

python utils/generate_audio_files.py

Download Kinetics-600 optical flow data (kinetics600flowmp4part) from [link](https://huggingface.co/datasets/hdong51/MultiOOD/tree/main) and extract (run `cat kinetics600_flow_mp4_part_ > kinetics600flowmp4.tar.gz and then tar -zxvf kinetics600flowmp4.tar.gz`).

Unzip all files and the directory structure should be modified to match:

Kinetics-600
├── video
|   ├── acting in play
|   |   ├── *.mp4
|   |   ├── *.wav
|   |── ...


├── flow
|   ├── acting in play
|   |   ├── *_flow_x.mp4
|   |   ├── *_flow_y.mp4
|   ├── ...

Dataset Splits

The splits for Multimodal Near-OOD and Far-OOD Benchmarks are provided in https://github.com/donghao51/MultiOOD under HMDB-rgb-flow/splits/ for HMDB51, UCF101, HAC, and Kinetics-600, and under EPIC-rgb-flow/splits/ for EPIC-Kitchens.

Methodology

<div style="text-align:left"> <img src="frame.jpg" width="70%" height="100%"> </div>

An overview of the proposed framework for Multimodal OOD Detection. We introduce A2D algorithm to encourage enlarging the prediction discrepancy across modalities. Additionally, we propose a novel outlier synthesis algorithm, NP-Mix, designed to explore broader feature spaces, which complements A2D to strengthen the OOD detection performance.

Contact

If you have any questions, please send an email to donghaospurs@gmail.com

Citation

If you find our work useful in your research please consider citing our paper:

@article{dong2024multiood,
	author   = {Hao Dong and Yue Zhao and Eleni Chatzi and Olga Fink},
	title    = {{MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities}},
    journal  = {arXiv preprint arXiv:2405.17419},
	year     = {2024},
}