Guo1115/SportMM-LT

SportMM-LT

English | 中文说明

English

SportMM-LT is a multimodal benchmark for evaluating long-tail sports knowledge in vision-language models.

It contains 421 image-question-answer samples across three sports domains:

• —basketball
• —football
• —table tennis

Dataset Overview

SportMM-LT is designed to evaluate whether vision-language models can answer fine-grained, domain-specific sports questions from images. The benchmark focuses on long-tail knowledge that is less likely to appear frequently in general-purpose pretraining data.

Each sample consists of:

• —an image
• —a question
• —a ground-truth answer
• —category annotations

File Structure

This dataset repository contains:

• —SportMM-LT.jsonl: the main benchmark file
• —category.json: category definitions
• —images/: image files organized by sport
• —images/basketball/
• —images/football/
• —images/table_tennis/

Data Format

Each line in SportMM-LT.jsonl is a JSON object with the following fields:

• —index: global sample index
• —id: sample identifier
• —sport: sport domain
• —question: question text
• —image: relative image path
• —answer: ground-truth answer
• —category: main category
• —subcategory: subcategory
• —hop: reasoning hop count
• —multiple: whether multiple answers are acceptable

Example:

{
  "index": 1,
  "id": "basketball_001",
  "sport": "basketball",
  "question": "What is the name of the athlete in the image? Provide only the name in English without any extra words or symbols.",
  "image": "images/basketball/001.jpg",
  "answer": "Ayomide Ajiboye",
  "category": "Identity and Organization",
  "subcategory": "Athlete Identification",
  "hop": 1,
  "multiple": false
}

Usage

Users can use this dataset to evaluate vision-language models on image-grounded sports question answering.

A typical evaluation workflow is:

1. 1.Download the dataset files.
2. 2.Run inference on each sample.
3. 3.Save model outputs in JSONL format.
4. 4.Compare predictions against the reference answers using an evaluation script.

An example prediction record may look like this:

{
  "index": 1,
  "id": "basketball_001",
  "sport": "basketball",
  "question": "What is the name of the athlete in the image? Provide only the name in English without any extra words or symbols.",
  "image": "images/basketball/001.jpg",
  "answer": "Ayomide Ajiboye",
  "category": "Identity and Organization",
  "subcategory": "Athlete Identification",
  "hop": 1,
  "multiple": false,
  "prediction": "Ayomide Ajiboye"
}

Notes

• —This dataset is intended for academic research and benchmark evaluation.
• —Some images may originate from third-party public websites or online resources.
• —The copyright of such images remains with their original rights holders.
• —Users are responsible for ensuring that their use of the dataset complies with applicable copyright, licensing, and data usage requirements.

Citation

The paper associated with SportMM-LT has been accepted, but the final publication information is not yet available. The citation entry will be updated after official publication.

中文说明

SportMM-LT 是一个用于评估视觉语言模型长尾体育知识能力的多模态 benchmark。

该数据集共包含 421 条图像-问题-答案样本，覆盖三个体育领域：

• —basketball
• —football
• —table tennis

数据集概述

SportMM-LT 旨在评估视觉语言模型是否能够根据图像回答细粒度、专业化的体育问题。该 benchmark 重点关注长尾知识，即那些不太可能在通用预训练数据中高频出现的体育专业知识。

每条样本包含：

• —一张图像
• —一个问题
• —一个标准答案
• —类别标注信息

文件结构

该数据集仓库包含：

• —SportMM-LT.jsonl：主数据集文件
• —category.json：类别定义文件
• —images/：按运动项目组织的图像文件
• —images/basketball/
• —images/football/
• —images/table_tennis/

数据格式

SportMM-LT.jsonl 中的每一行都是一个 JSON 对象，包含以下字段：

• —index：全局样本编号
• —id：样本标识符
• —sport：所属体育领域
• —question：问题文本
• —image：图像相对路径
• —answer：标准答案
• —category：一级类别
• —subcategory：二级类别
• —hop：推理跳数
• —multiple：是否接受多个答案

示例：

{
  "index": 1,
  "id": "basketball_001",
  "sport": "basketball",
  "question": "What is the name of the athlete in the image? Provide only the name in English without any extra words or symbols.",
  "image": "images/basketball/001.jpg",
  "answer": "Ayomide Ajiboye",
  "category": "Identity and Organization",
  "subcategory": "Athlete Identification",
  "hop": 1,
  "multiple": false
}

使用方式

用户可以使用该数据集评估视觉语言模型在图像驱动体育问答任务上的表现。

一个典型的评测流程如下：

1. 1.下载数据集文件。
2. 2.对每条样本运行模型推理。
3. 3.将模型输出保存为 JSONL 格式。
4. 4.使用评测脚本将预测结果与参考答案进行比较。

预测文件中的单条记录可以写成如下形式：

{
  "index": 1,
  "id": "basketball_001",
  "sport": "basketball",
  "question": "What is the name of the athlete in the image? Provide only the name in English without any extra words or symbols.",
  "image": "images/basketball/001.jpg",
  "answer": "Ayomide Ajiboye",
  "category": "Identity and Organization",
  "subcategory": "Athlete Identification",
  "hop": 1,
  "multiple": false,
  "prediction": "Ayomide Ajiboye"
}

说明

• —该数据集主要用于学术研究和 benchmark 评测。
• —其中部分图像可能来源于第三方公开网站或在线资源。
• —相关图像版权仍归原始权利人所有。
• —使用者需自行确认其使用方式符合适用的版权、许可和数据使用要求。

引用

SportMM-LT 对应论文已经接收，但正式出版信息尚未发布。待论文正式上线后，将补充更新 citation。