hf-audio/open-asr-leaderboard-moss-transcribe-diarize

MOSS-Transcribe-Diarize Open ASR evaluator

This Space is the MOSS-specific evaluator used by the same HF Jobs workflow as the Musci-ASR submission. It keeps one standard Open ASR manifest per dataset and uses the official MOSS remote-code processor.

The current public leaderboard submission uses these seven config/split pairs:

• —ami_cleaned/test
• —earnings22/test
• —gigaspeech_cleaned/test
• —librispeech/test.clean
• —librispeech/test.other
• —spgispeech/test
• —voxpopuli_cleaned_aa/test

The local canonical manifests are indexed by data/openasrleaderboard_latest/dataset_index.json.

The generated timestamp and speaker markup is removed before writing pred_text; the original model output is not used as a WER token.

run_eval.py performs true batched processor and generation calls. The validated H200 submission setting is batch size 256 with one warmup batch. Batched decoding is capped at 512 generated tokens; any sample that reaches its duration-based safety budget is forced to EOS so one pathological sample cannot stall the other 255 samples in its batch. The budget is deliberately generous for short-form English audio (12 tokens per second plus 64 tokens, with a minimum of 128 and maximum of 512).

Validated on the cleaned seven-set benchmark at dataset revision b6bdcd0beb34f8975dc659796176d88f43aff502:

• —AVG7 WER: 5.17%
• —aggregate RTFx: 294.02
• —batch size: 256
• —model revision: e5118b411bf5a77d7a90c4941066bec93c967312
• —public result bucket: Cb1ock/open-asr-leaderboard-moss-transcribe-diarize-results