LeeHarrold/musiccaps-mot-tokens

MusicCaps Pre-Encoded Tokens for Mixture-of-Transformers (MoT)

Dataset Description

This dataset contains pre-encoded audio tokens from the MusicCaps dataset, processed through Meta's MusicGen EnCodec tokenizer for use in Mixture-of-Transformers (MoT) training.

Dataset Summary

• —5,233 music clips encoded as discrete tokens
• —4 codebook layers from MusicGen's EnCodec
• —~500 tokens per 10-second clip
• —Compressed from ~12GB audio to 82MB tokens
• —Ready for multimodal language model training

Intended Use

This dataset is designed for:

• —Training Mixture-of-Transformers (MoT) models that combine Llama and MusicGen
• —Research in multimodal language models with audio understanding
• —Experiments in music captioning and generation
• —Efficient training without on-the-fly audio encoding

Dataset Structure

Data Fields

• —ytid: YouTube video ID
• —caption: Human-written music description
• —aspect_list: Musical aspects mentioned in caption
• —audioset_positive_labels: AudioSet labels
• —audio_codes: Pre-encoded tokens shape [4, ~500] (4 codebooks, ~500 time steps)
• —n_codebooks: Number of codebooks (always 4)
• —seq_length: Sequence length of tokens
• —start_s: Start time in original video
• —end_s: End time in original video
• —author_id: Caption author ID
• —is_balanced_subset: Whether part of balanced subset
• —is_audioset_eval: Whether part of AudioSet eval

Data Splits

• —train: 4,710 examples (90%)
• —test: 523 examples (10%)

Pre-Encoding Details

Encoding Process

1. 1.Audio Loading: 10-second clips from MusicCaps
2. 2.Resampling: All audio resampled to 32kHz (MusicGen requirement)
3. 3.Tokenization: MusicGen EnCodec with 4 codebooks @ 50Hz
4. 4.Vocabulary: 2048 tokens per codebook
5. 5.Compression: ~12GB audio → 82MB tokens

Token Format

# Shape: [4, ~500]
# - 4 codebooks (hierarchical encoding)
# - ~500 time steps (50Hz * 10 seconds)
# Each value in range [0, 2047]

Usage

Loading the Dataset

from datasets import load_dataset

dataset = load_dataset("YOUR_USERNAME/musiccaps-mot-tokens")

# Access pre-encoded tokens
sample = dataset['train'][0]
audio_codes = np.array(sample['audio_codes'])  # Shape: [4, ~500]
caption = sample['caption']

Using with MoT Training

# Shift tokens for combined vocabulary
# Llama uses tokens [0, 128255]
# Audio uses tokens [128256, 130303]
audio_tokens = audio_codes + 128256

# Interleave codebooks for sequence modeling
# [c0_t0, c1_t0, c2_t0, c3_t0, c0_t1, ...]
b, k, t = 1, audio_codes.shape[0], audio_codes.shape[1]
interleaved = audio_codes.transpose(1, 0).reshape(-1)

Training Configuration

Recommended settings for MoT adapter training:

• —Model: Llama 3.2 1B + MusicGen Small
• —Adapter dims: 67M parameters
• —Batch size: 8-16 (on GPU)
• —Learning rate: 1e-4
• —Max sequence: 1024 tokens (32 text + 992 audio)

Citation

If you use this dataset, please cite:

@dataset{musiccaps_mot_tokens,
  title={MusicCaps Pre-Encoded Tokens for MoT},
  author={Your Name},
  year={2024},
  publisher={HuggingFace}
}

@article{musiccaps,
  title={MusicCaps: Music Audio Captioning with Text-Audio Retrieval},
  author={Agostinelli et al.},
  year={2023}
}

Acknowledgments

• —Google for the original MusicCaps dataset
• —Meta for MusicGen and EnCodec
• —The Mixture-of-Transformers paper authors

License

This dataset inherits licenses from:

• —MusicCaps: [Research use]
• —Encoded representations are derivative works

Please ensure compliance with original dataset licenses.