Datasetpublic

matheusfpinto/test_tokenization

Orpheus PT-BR SNAC 8192 Data Fields input_ids: List[int] length 8192 attention_mask: List[int] length 8192 metadata: Dict[str, Any] containing original dataset, config, split, audio_length, text Usage from datasets import load_dataset ds = load_dataset("matheusfpinto/orpheus-ptbr-snac-8192", split="train", streaming=True) sample = next(iter(ds)) assert len(sample["input_ids"]) == 8192 Citation Please cite the original… See the full description on the dataset page: https://huggingface.co/datasets/matheusfpinto/test_tokenization.

sourceHugging Faceupdated 11mo agoView on Hugging Face

1likes9downloads

Dataset Card

Orpheus PT-BR SNAC 8192

Dataset Description

—Source: opedromartins/ASR-datasets-ptbr (all configs)
—Processing: text tokenization with canopylabs/orpheus-tts-0.1-pretrained, SNAC 24k audio codes (hubertsiuzdak/snac_24khz)
—Sequence Format: [text_token_ids ... <|eot_id|> ... snac_audio_ids]
—Sequence Length: 8192 tokens (padded)

Data Fields

—input_ids: List[int] length 8192
—attention_mask: List[int] length 8192
—metadata: Dict[str, Any] containing original dataset, config, split, audio_length, text

Usage

python

from datasets import load_dataset

ds = load_dataset("matheusfpinto/orpheus-ptbr-snac-8192", split="train", streaming=True)
sample = next(iter(ds))
assert len(sample["input_ids"]) == 8192

Citation

Please cite the original data sources and Orpheus/SNAC works where appropriate. --- prettyname: Test Tokenization taskids:

—text-to-speech language:
—pt ---