yanbing2/RacketVision
RacketVision Dataset RacketVision is a large-scale, multi-sport dataset and benchmark for advancing computer vision in sports analytics, covering badminton, table tennis, and tennis. It is the first dataset to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. The benchmark tackles three interconnected tasks: fine-grained ball tracking, articulated racket pose… See the full description on the dataset page: https://huggingface.co/datasets/yanbing2/RacketVision.
02k
1---2license: mit3language:4 - en5pretty_name: RacketVision Dataset6size_categories:7 - 10K<n<100K8task_categories:9 - object-detection10 - video-classification11tags:12 - sports-analytics13 - computer-vision14 - object-tracking15 - trajectory-prediction16 - ball-tracking17 - racket-pose-estimation18 - badminton19 - table-tennis20 - tennis21 - racket-sports22---23 24# RacketVision Dataset25 26[](https://arxiv.org/abs/2511.17045)27[](https://aaai.org/)28[](https://github.com/OrcustD/RacketVision/)29[](https://huggingface.co/datasets/linfeng302/RacketVision)30[](https://huggingface.co/linfeng302/RacketVision-Models)31 32**RacketVision** is a large-scale, multi-sport dataset and benchmark for advancing computer vision in sports analytics, covering **badminton**, **table tennis**, and **tennis**. It is the first dataset to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. The benchmark tackles three interconnected tasks: fine-grained **ball tracking**, articulated **racket pose estimation**, and predictive ball **trajectory forecasting**.33 3435 36## Using this Hub repository37 38This dataset is distributed as **static files** (videos, CSV, JSON, PKL). Download it with the Hugging Face CLI, then follow the [project README](https://github.com/OrcustD/RacketVision/blob/main/README.md) for environment setup and training:39 40```bash41# Official code layout (clone https://github.com/OrcustD/RacketVision ): from repo root42hf download linfeng302/RacketVision --repo-type dataset --local-dir source/data43 44# Stand-alone data folder only (you must point module configs or --data_root to this directory)45hf download linfeng302/RacketVision --repo-type dataset --local-dir data46```47 48The in-browser Dataset Viewer may not fully load all assets: COCO detection and pose JSON files use different annotation schemas, so they are not merged into a single `datasets`-style table. Use the files on disk as documented below.49 50## Directory Layout51 52```53data/54├── annotations/55│ └── dataset_info.json # Global dataset metadata (clip list, splits)56│57├── info/ # COCO-format annotations for RacketPose58│ ├── train_det_coco.json # Detection: bbox annotations (train split)59│ ├── val_det_coco.json60│ ├── test_det_coco.json61│ ├── train_pose_coco.json # Pose: keypoint annotations (train split)62│ ├── val_pose_coco.json63│ └── test_pose_coco.json64│65├── <sport>/ # badminton / tabletennis / tennis66│ ├── videos/67│ │ └── <match>_<rally>.mp4 # Raw video clips68│ ├── all/69│ │ └── <match>/70│ │ ├── csv/<rally>_ball.csv # Ball ground truth annotations71│ │ └── racket/<rally>/*.json # Racket ground truth annotations72│ ├── interp_ball/ # Interpolated ball trajectories (for rebuilding TrajPred data)73│ ├── merged_racket/ # Merged racket predictions (for rebuilding TrajPred data)74│ └── info/75│ ├── metainfo.json # Sport-specific metadata76│ ├── train.json # [[match_id, rally_id], ...] for training77│ ├── val.json # Validation split78│ └── test.json # Test split79│80└── data_traj/ # Pre-built trajectory prediction datasets81 ├── ball_racket_<sport>_h20_f5.pkl # Short-horizon: 20 history → 5 future82 └── ball_racket_<sport>_h80_f20.pkl # Long-horizon: 80 history → 20 future83```84 85**Local preprocessing (required for BallTrack):** after download, generate per-match `frame/<rally>/` (JPG frames) and `median.npz` from the videos using `DataPreprocess/extract_frames.py` and `DataPreprocess/create_median.py`. These are omitted from the Hub release to save space; see the [project README](https://github.com/OrcustD/RacketVision/blob/main/README.md).86 87## Data Formats88 89### Ball Annotations (`csv/<rally>_ball.csv`)90 91| Column | Type | Description |92|--------|------|-------------|93| Frame | int | 0-indexed frame number |94| X | int | Ball center X in pixels (1920×1080) |95| Y | int | Ball center Y in pixels |96| Visibility | int | 1 = visible, 0 = not visible |97 98### Racket Annotations (`racket/<rally>/<frame_id>.json`)99 100Per-frame JSON with a list of racket instances, each containing:101 102```json103{104 "category": "badminton_racket",105 "bbox_xywh": [x, y, w, h],106 "keypoints": [[x1, y1, vis], [x2, y2, vis], ...]107}108```109 110**5 keypoints** are defined: `top`, `bottom`, `handle`, `left`, `right`.111 112### COCO Annotations (`info/*_coco.json`)113 114Standard COCO format used by RacketPose for training/evaluation:115 116- **Detection** (`*_det_coco.json`): 3 categories — `badminton_racket`, `tabletennis_racket`, `tennis_racket`.117- **Pose** (`*_pose_coco.json`): 1 category (`racket`) with 5 keypoints.118 119### Trajectory PKL (`data_traj/*.pkl`)120 121Pickle files containing pre-processed sliding-window samples. Each PKL has:122 123```python124{125 'train_samples': [...], # List of sample dicts126 'test_samples': [...],127 'train_dataset': ..., # Legacy Dataset objects128 'test_dataset': ...,129 'metadata': {130 'history_len': 80,131 'future_len': 20,132 'sports': ['badminton'],133 'total_samples': N,134 'train_size': ...,135 'test_size': ...136 }137}138```139 140Each sample dict:141 142```python143{144 'history': np.array(shape=(H, 2)), # Normalised [X, Y] in [0, 1]145 'future': np.array(shape=(F, 2)),146 'history_rkt': np.array(shape=(H, 10)), # 5 keypoints × 2 coords, normalised147 'future_rkt': np.array(shape=(F, 10)),148 'sport': str,149 'match': str,150 'sequence': str,151 'start_frame': int152}153```154 155**Normalisation**: Ball coordinates are divided by (1920, 1080). Racket keypoints are divided by the same values.156 157### Split Files (`<sport>/info/train.json`)158 159JSON list of `[match_id, rally_id]` pairs:160 161```json162[["match1", "000"], ["match1", "001"], ...]163```164 165## Generating Data from Scratch166 167If you have the raw videos, use `DataPreprocess/` scripts in the [code repository](https://github.com/OrcustD/RacketVision/) to prepare all intermediate files:168 169```bash170cd DataPreprocess171 172# 1. Extract video frames to JPG173python extract_frames.py --data_root ../data --sport badminton174 175# 2. Compute median background frame176python create_median.py --data_root ../data --sport badminton177 178# 3. Generate dataset_info.json and per-sport split files179python generate_dataset_info.py --data_root ../data180 181# 4. Generate COCO annotations for RacketPose182python generate_coco_annotations.py --data_root ../data183```184 185## Generating Trajectory Data186 187After running BallTrack and RacketPose inference, build `data_traj/` PKLs:188 189```bash190cd TrajPred191 192# Interpolate short gaps in ball predictions193python linear_interpolate_ball_traj.py --data_root ../data --sport badminton194 195# Merge racket predictions with ground truth annotations196python merge_gt_with_predictions.py --data_root ../data --sport badminton197 198# Build PKL dataset199python build_dataset.py --data_root ../data --sport badminton --history 80 --future 20200```201 202## Citation203 204If you find this work useful, please consider citing:205 206```bibtex207@inproceedings{dong2026racket,208 title={Racket Vision: A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis},209 author={Dong, Linfeng and Yang, Yuchen and Wu, Hao and Wang, Wei and Hou, Yuenan and Zhong, Zhihang and Sun, Xiao},210 booktitle={Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},211 year={2026}212}213```214