BoJack/Omni-Cloze

Omni-Cloze Benchmark

**📖 Paper** | **🕵️ Omni-Detective Pipeline** | **🧑‍🏫 Omni-Cloze Benchmark**

Guides

Omni-Cloze frames detailed captioning evaluation as a cloze-style multiple-choice proxy task. Omni-Cloze is a unified benchmark for evaluating detailed captioning across audio-only, visual-only, and audio–visual settings. The dataset spans 9 main domains and 47 sub-categories covering diverse topics such as education, entertainment, sports, news, science, and lifestyle, with a total of 2k video clips with 70k fine-grained cloze blanks.

[image]

Quick Start

1. Prepare Video Data

The video dataset is split into several tarball parts. Concatenate and extract them to the videos/ directory:

for f in videos.part*.tar; do tar -xvf "$f"; done

2. Prepare Inference Results

The core metadata file is omni_cloze.jsonl, which contains 2,320 audio-visual files and their corresponding cloze questions.

To evaluate your model, you must first run inference and save the generated descriptions into a new field named `predicted_caption` within the JSONL file. Each line in your input file should follow this structure:

{
  "uuid": 1,
  "video_path": "./videos/0000001.mp4",
  "predicted_caption": "Your model's detailed description of the audio and visual content goes here...",
  "...": [...] 
}

3. Run Evaluation

The evaluation process uses an LLM to map your detailed captions to the specific cloze blanks.

# 1. Set API environment variables
export OPENAI_API_KEY="your-api-key-here"
export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"

# 2. Specify input and output data paths, and the model predicted caption field name is "predicted_caption" in the input file
input_file="your-input-file-here.jsonl"
output_file="your-output-file-here.jsonl"

# 3. Run the evaluation script
python generate_prediction.py --input $input_file --output $output_file --workers 100

# 4. Run the statistics script
python compute_acc.py --input $output_file --show-subcategory

Citation

If you find Omni-Cloze useful for your research, please consider citing our work:

@article{omni-captioner,
  title={Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception},
  author={Ma, Ziyang and Xu, Ruiyang and Xing, Zhenghao and Chu, Yunfei and Wang, Yuxuan and He, Jinzheng and Xu, Jin and Heng, Pheng-Ann and Yu, Kai and Lin, Junyang and others},
  journal={Proc. ICLR},
  year={2026}
}