arda-argmax/fastmss-debug-v0.5

FastMSS synthetic multi-speaker meetings - parquet edition

Streaming-friendly parquet shards of the FastMSS synthetic multi-speaker conversational corpus. Each row is one mixture with the audio bytes embedded inline (16 kHz mono WAV) plus per-segment diarization timestamps, per-word transcript and the full lhotse cut as a JSON blob. See `fastmss/hf_dataset.py` for the schema docstring.

Subsets and splits

• —`debug_v0.5` — splits: train — 10 mixtures, 20.5 min total, 39 unique speakers, 1 shard(s) (40.0 MB).

Layout

<subset>/
    data/
        train-XXXXX-of-YYYYY.parquet
        val-XXXXX-of-YYYYY.parquet     # if subsplit
        split_assignment.json          # if subsplit
    provenance/
        all_cuts.jsonl.gz   # source utterance pool
        all_rooms.json      # RIR pool metadata
        noise_files.txt     # background noise pool
        sim.log             # generator log

Per-row schema

Loading

With the YAML configs block above, HF datasets exposes each subset as a config and the train/val shards as proper splits:

from datasets import load_dataset

# whole subset (default = train split):
ds = load_dataset("<user-or-org>/<repo-name>", "v0.1")

# explicit split:
train = load_dataset("<user-or-org>/<repo-name>", "v0.1", split="train")
val   = load_dataset("<user-or-org>/<repo-name>", "v0.1", split="val")

# streaming:
stream = load_dataset(
    "<user-or-org>/<repo-name>", "v0.1", split="train", streaming=True
)
for sample in stream:
    sample["audio"]["array"]        # decoded float32 waveform
    sample["timestamps_start"]      # diarization segment starts
    sample["timestamps_end"]        # diarization segment ends
    sample["speakers"]              # one label per segment
    sample["transcript"]            # word tokens
    sample["word_speakers"]         # per-word speakers

Drop the lhotse JSON blob if you don't need it:

ds = ds.remove_columns(["manifest_json"])

Rebuild a lhotse CutSet from any subset:

import json
from lhotse import CutSet, MonoCut
cuts = CutSet.from_cuts(
    MonoCut.from_dict(json.loads(s["manifest_json"])) for s in ds
)

Generating an HF-compatible dataset from scratch

The generation pipeline lives in the FastMSS repo. It produces lhotse manifests + audio first, then converts them into the parquet layout shipped here. Reproduce a subset with:

1. Synthesize the lhotse split — mixes utterances + RIRs + noise into `<dataset_root>/<subset>/ with audio/, manifests/, rttm_word/ and rttm_segment/` subfolders.

# Adjust config_name / dataset_root for the subset you want
python sim.py \
    --config-path config/table1 --config-name datagen_v0.1 \
    output_dir=generated_dataset/v0.1

2. Convert to streamable parquet — writes one parquet shard per `--shard-size mixtures, embedding WAV bytes inline and computing every column above. The default --parquet-batch-size 64 keeps row groups small enough for the Hugging Face dataset viewer on long-audio subsets. --subsplits` performs a deterministic train/val split with a reproducible seed.

python scripts/convert_to_parquet.py \
    --dataset-root generated_dataset \
    --output-root  generated_dataset_parquet \
    --splits v0.1 \
    --subsplits train:800,val:200 \
    --subsplit-seed 42 \
    --shard-size 256 \
    --parquet-batch-size 64

# Smaller subset that doesn't need a train/val split (e.g. debug):
python scripts/convert_to_parquet.py \
    --dataset-root generated_dataset \
    --output-root  generated_dataset_parquet \
    --splits debug

3. Upload to the Hub — stages a `<subset>/data/ + <subset>/provenance/ layout, generates this README's YAML configs: block automatically, and pushes via HfApi.upload_large_folder` (resumable / parallel).

hf auth login   # or set HF_TOKEN

python scripts/upload_parquet_to_hf.py \
    --repo-id <user-or-org>/<dataset-name> \
    --parquet-root generated_dataset_parquet \
    --dataset-root generated_dataset

Useful flags:

• —`--splits debug v0.1` — push only some subsets
• —`--private` — only honored on first repo create
• —`--dry-run` — stage the layout to a temp dir and print it without contacting the Hub
• —`--no-provenance — skip the provenance/` sidecars

4. Verify the round-trip locally:

pytest tests/test_hf_parquet_conversion.py

These tests build a synthetic FastMSS split in a tmp dir, run the converter, and assert byte-for-byte equivalence between the lhotse manifests/RTTM/audio and the parquet rows (including a `json.loads(row['manifest_json']) == cut` round-trip and a deterministic-shuffle subsplit check).

See `fastmss/hf_dataset.py` for the per-row schema and helper API; both scripts above are thin CLI wrappers over it.