SignerX/SignVerse-2M

<div align="center" style="position: relative; width: 100%; max-width: 1400px; margin: 0 auto 24px auto;"> <img src="https://signerx.github.io/SignVerse-2M/static/images/backgroundgallerybig.gif" alt="SignVerse-2M cover" style="width: 100%; display: block; border-radius: 18px;" /> <div style=" position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; background: linear-gradient(to bottom, rgba(0,0,0,0.18), rgba(0,0,0,0.28)); border-radius: 18px; "

<div style=" color: white; font-size: 3.4rem; font-weight: 800; letter-spacing: 0.04em; text-shadow: 0 4px 20px rgba(0,0,0,0.45); "

SignVerse-2M </div> </div> </div>

SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languages

Links: [[Paper]](https://arxiv.org/abs/2605.01720) | [[Data Files]](https://huggingface.co/datasets/SignerX/SignVerse-2M/tree/main/dataset) | [[Project Page]](https://signerx.github.io/SignVerse-2M)

SignVerse-2M is a large-scale multilingual pose-native dataset for sign language research. The dataset reorganizes publicly available sign language videos into a unified DWPose-based representation and releases the result as approximately 2 million clips from 39,196 videos covering 55+ sign languages. Rather than distributing raw RGB video, SignVerse-2M provides per-frame body, hand, and face keypoints together with structured subtitle supervision, making the corpus directly usable for pose-conditioned sign language generation, recognition, and translation research.

Overview

Existing large-scale sign language resources are typically organized as video-text corpora. That format is appropriate for RGB-based recognition or translation, but it is not the most natural interface for modern pose-driven generation pipelines, which increasingly operate on standardized human keypoint controls such as DWPose. SignVerse-2M addresses this mismatch by converting multilingual public sign language videos into a common pose space.

The release is intended to support research questions such as:

• —multilingual sign language generation in pose space
• —pose-based sign language recognition and translation
• —cross-lingual transfer across heterogeneous sign language sources
• —benchmarking of sign language motion representations under open-world conditions

Key Characteristics

Why A Pose-Native Release

SignVerse-2M should not be understood as merely a larger multilingual video-text corpus. Its main contribution is the release of a unified pose-native interface for sign language research.

Compared with raw-video releases, the pose-native representation offers three practical advantages:

1. 1.It reduces nuisance variation from background, clothing, and appearance, allowing models to focus more directly on motion.
2. 2.It aligns naturally with contemporary pose-conditioned generation pipelines that already consume DWPose-like controls.
3. 3.It provides a common representation for multilingual benchmarking, making comparisons across methods more interpretable.

Data Source And Processing

The corpus is built from publicly available multilingual sign language videos (YouTube-SL-25), including resources inherited from large public sign language collections such as YouTube-SL-55 and related open web sources. Each video is processed through a unified pipeline that:

1. 1.retrieves metadata and available subtitles,
2. 2.structures subtitle tracks into segment-level and document-level text,
3. 3.decodes the video at 24 FPS,
4. 4.applies DWPose to extract body, hand, and face keypoints frame by frame,
5. 5.packages the outputs into per-video artifacts for public release.

No manual keypoint annotation is provided. The keypoints and subtitles are produced automatically through the preprocessing pipeline.

Languages

The corpus covers more than 55 sign languages. The current release spans 55 language identifiers:

The language distribution is long-tailed rather than balanced. High-resource languages account for a disproportionate share of the total data volume.

Repository Structure

The public release is organized around .tar shards stored under dataset/. Each shard contains per-video directories:

dataset/
  Sign_DWPose_NPZ_000001.tar
  Sign_DWPose_NPZ_000002.tar
  ...

Within each shard:

{video_id}/
  poses.npz
  caption.json
  {video_id}.complete

The main files are:

• —poses.npz: per-video DWPose payload with frame-wise keypoints
• —caption.json: structured subtitle and supervision metadata
• —.complete: completion marker produced by the processing pipeline

Data Schema

poses.npz

Each poses.npz file stores a person-centric per-frame representation. A simplified schema is shown below:

{
    "video_id": str,
    "fps": float,
    "num_frames": int,
    "frame_ids": int[T],
    "width": int,
    "height": int,
    "frames": [
        {
            "num_people": int,
            "frame_id": int,
            "width": int,
            "height": int,
            "person_0": {
                "body": float[18, 3],
                "face": float[68, 3],
                "left_hand": float[21, 3],
                "right_hand": float[21, 3],
            },
            # optional additional people:
            # "person_1": { ... }
        },
        ...
    ]
}

Keypoint coordinates are stored in pixel space as (x, y, score), where confidence scores lie in [0, 1].

caption.json

{
  "video_id": "...",
  "sign_language": "ase",
  "title": "...",
  "duration_s": 312.4,
  "segments": [
    { "start": 0.0, "end": 4.2, "text": "..." }
  ],
  "document_text": "...",
  "english_source": "native"
}

The field english_source records whether the English supervision is native or automatically selected from an available translated subtitle track.

Loading Example

import json
import tarfile
import numpy as np

with tarfile.open("dataset/Sign_DWPose_NPZ_000001.tar") as tar:
    tar.extractall("./tmp_signverse")

npz = np.load("./tmp_signverse/{video_id}/poses.npz", allow_pickle=True)
frames = npz["frames"].tolist()
body = frames[0]["person_0"]["body"]

with open("./tmp_signverse/{video_id}/caption.json", "r", encoding="utf-8") as f:
    caption = json.load(f)

print(body.shape)
print(caption["segments"][0]["text"])

Visualization And Reproduction

The repository includes scripts for inspecting the released pose files and for reproducing the processing pipeline.

Visualize one pose file

python scripts/visualize_dwpose_npz.py \
    --npz extracted/{video_id}/poses.npz \
    --style openpose \
    --out viz/

Reproduce the pipeline

# Single machine
bash reproduce_independently.sh

# SLURM cluster
bash reproduce_independently_slurm.sh

The pipeline is organized into acquisition, subtitle structuring, pose extraction, and upload/publication stages.

Benchmark Setting

The accompanying paper introduces a multilingual text-to-pose benchmark for sign language generation. A generated DWPose sequence is evaluated through back-translation into spoken text, and standard text metrics such as BLEU and ROUGE are reported against the source input. The benchmark repository also provides a SignDW Transformer baseline in both small and large model configurations.

For model code and experimental setup, refer to the benchmark repository:

• —https://github.com/SignerX/SignVerse-2M

Intended Use

The release is intended for research use, including:

• —sign language generation from text via pose space
• —pose-based sign language translation and recognition
• —cross-lingual transfer, adaptation, and benchmarking
• —comparison of pose-native motion representations under open-world distributions

The release is not intended for:

• —safety-critical interpretation in medical, legal, or emergency settings
• —re-identification of individual signers
• —claims of full linguistic coverage for any specific sign language

Data Governance

Exclusion / Takedown Mechanism

SignVerse-2M indexes all samples strictly by video_id and does not re-host or redistribute original video content. If a source video is deleted or made private by its creator, no copy exists in our release to propagate — access to any derived content tied to that video_id is automatically severed.

Creators may additionally request explicit removal of their corresponding pose data by contacting issue link. Verified removal requests will be processed within 7 business days, and an exclusion log will be maintained and version-tracked alongside future dataset releases.

Face-Landmark-Excluded Variant

For downstream applications that do not require facial keypoints, a variant of the dataset with facial landmarks excluded is available at link, reducing the identity-bearing channels retained in the released data.

<!--

Limitations

Users should account for the following limitations when interpreting results:

• —Keypoints are extracted automatically and may be noisy under fast motion, occlusion, or multi-person scenes.
• —Fine-grained handshape distinctions are only partially captured by the released 21-keypoint hand representation.
• —Non-manual linguistic signals such as facial expression and mouthing are only partially represented by 68 face landmarks.
• —Subtitle timing and translations are automatically processed and may contain alignment or semantic errors.
• —The corpus is language-imbalanced and inherits the long-tail distribution of public web video sources.
• —person_0 is treated as the primary signer, which may be imperfect in multi-signer videos. -->

Responsible Use

SignVerse-2M is derived from publicly posted sign language videos. This repository does not redistribute raw RGB videos; it releases pose keypoints and structured subtitle text only. Even so, pose sequences may still carry information that can contribute to signer identification when combined with external metadata. Users should treat the corpus as human-subject-derived data and use it responsibly.

Re-identification of individual signers is explicitly listed as an unintended use of this dataset (see Intended Use above). Users are expected to comply with applicable data protection regulations in their jurisdiction when using this resource.

The data distribution is also shaped by what is publicly available online. Educational or interpreter-style content may be overrepresented, while conversational, regional, or community-specific signing practices may be underrepresented.

Citation

If you use SignVerse-2M in academic work, please cite:

@misc{fang2026signverse2mtwomillionclipposenativeuniverse,
      title={SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languages}, 
      author={Sen Fang and Hongbin Zhong and Yanxin Zhang and Dimitris N. Metaxas},
      year={2026},
      eprint={2605.01720},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.01720}, 
}

License

The released dataset annotations, pose keypoints, and accompanying metadata are distributed under CC BY-NC 4.0.

Source videos are not redistributed in this repository and remain subject to the original platform terms and the rights of their respective creators.

CC BY-NC 4.0 applies exclusively to the derived DWPose keypoint annotations and structured subtitle metadata produced by our processing pipeline. Original video content remains governed by its source platform's Terms of Service and the rights of the original creator; no rights over the underlying video are claimed or conveyed by this release.