pr0mila-gh0sh/MediBeng-FL

MediBeng-FL

A Federated Learning–Ready Extension of [MediBeng](https://huggingface.co/datasets/pr0mila-gh0sh/MediBeng)

MediBeng-FL augments the original MediBeng dataset with rich, realistic synthetic metadata designed to benchmark federated learning (FL) algorithms on clinical Bengali-English code-switched speech.

Every sample retains the original six MediBeng columns — audio, text, translation, speaker_name, utterance_pitch_mean, utterance_pitch_std — and adds 9 new FL-metadata columns that simulate the heterogeneity found in real multi-hospital deployments.

Why a Separate Dataset?

MediBeng is a carefully curated clinical ASR dataset. MediBeng-FL layers in deployment context metadata: which hospital collected the audio, on what device, from which demographic group, and with how much background noise. This metadata enables controlled IID vs. Non-IID federated learning experiments without modifying the original benchmark.

Dataset Structure

Original Columns (unchanged from MediBeng)

New FL-Metadata Columns

Splits

Heterogeneity Design

Hospital → Device Anchor

Each hospital is anchored to a primary recording device (70 % of samples from that hospital use the anchor device):

SNR by Device Type (mean ± std dB)

Four Non-IID Split Strategies

All strategies produce 4 client shards (matching NUM_CLIENTS = 4).

1. Speaker Non-IID (noniid_speaker)

Split by speaker_gender:

• —Clients 0–1 → male utterances (two equal halves)
• —Clients 2–3 → female utterances (two equal halves)

2. Hospital Non-IID (noniid_hospital)

Group hospitals into pairs:

• —Client 0 → H0, H1 (clinical recorder, urban)
• —Client 1 → H2, H3 (telehealth)
• —Client 2 → H4, H5 (smartphone, community)
• —Client 3 → H6, H7 (desktop/tablet, mixed)

3. Device Non-IID (noniid_device)

Split by device_type:

• —Client 0 → clinical_recorder
• —Client 1 → telehealth
• —Client 2 → smartphone
• —Client 3 → desktop_mic + tablet

4. Demographic Non-IID (noniid_demographic) ← recommended

Split by client_id (accent-region cluster, richest heterogeneity):

• —Client 0 → Dhaka + Chittagong
• —Client 1 → Sylhet + Khulna
• —Client 2 → Rajshahi + Barisal
• —Client 3 → Rangpur + Mymensingh

Quick Start

from datasets import load_dataset

ds = load_dataset("pr0mila-gh0sh/MediBeng-FL")
train, test = ds["train"], ds["test"]

# IID partition (shuffle + split by index)
import numpy as np
idx = np.random.permutation(len(train))
k   = len(idx) // 4
client_shards = [idx[i*k:(i+1)*k].tolist() for i in range(4)]

# Non-IID partition — demographic axis (recommended)
from collections import defaultdict
noniid_shards = defaultdict(list)
for i, ex in enumerate(train):
    noniid_shards[ex["client_id"]].append(i)

# Access a shard
client_0 = train.select(noniid_shards[0])
print(client_0[0])

Reproducibility

All synthetic metadata was generated with a fixed seed (seed=42). The assignment logic is deterministic and documented in medibeng_synthetic_builder.py.

Related Resources

Citation

If you use MediBeng-FL in your research, please cite both the original MediBeng dataset and this extension:

@misc{promila_ghosh_2025,
    author       = { Promila Ghosh },
    title        = { MediBeng (Revision b05b594) },
    year         = 2025,
    url          = { https://huggingface.co/datasets/pr0mila-gh0sh/MediBeng },
    doi          = { 10.57967/hf/5187 },
    publisher    = { Hugging Face }
}

@dataset{medibeng_fl_2025,
    title     = {MediBeng-FL: A Federated Learning–Ready Extension of MediBeng},
    author    = {Promila Ghosh},
    year      = 2025,
    publisher = {Hugging Face},
    url       = {https://huggingface.co/datasets/pr0mila-gh0sh/MediBeng-FL},
    note      = {Extends pr0mila-gh0sh/MediBeng with synthetic FL metadata
                 (hospital, device, demographic, speaker heterogeneity).
                 Generate a DOI from the dataset page on Hugging Face.}
}

License

CC-BY-4.0 — same as the source MediBeng dataset. Free to use for research and commercial purposes with attribution.