CoolFace
Datasetpublic

sophia8888/clipquill-asr-benchmark

Measuring whisper-tiny vs whisper-base in a browser tab Word error rate, wall-clock timing, transfer size and peak memory for two quantised Whisper tiers running entirely client-side in a real Chrome window, with the scripts that produced every number. If you are building an in-browser transcription page, the two results worth knowing before you pick a model tier: On clean synthetic audio the two tiers tie. If that is all you test, you will conclude the tier does not matter… See the full description on the dataset page: https://huggingface.co/datasets/sophia8888/clipquill-asr-benchmark.

sourceHugging Facecc-by-4.0updated 8d agoView on Hugging Face
0likes142downloads
Dataset Card

Measuring whisper-tiny vs whisper-base in a browser tab

Word error rate, wall-clock timing, transfer size and peak memory for two quantised Whisper tiers running entirely client-side in a real Chrome window, with the scripts that produced every number.

If you are building an in-browser transcription page, the two results worth knowing before you pick a model tier:

  1. 1.On clean synthetic audio the two tiers tie. If that is all you test, you will conclude the tier does not matter, and you will be wrong. On real speech with noise, tiny collapses: weighted WER 59.0 % against base at 19.7 %.
  2. 2."Smaller model is faster" does not hold past about 13 seconds of audio. On byte-identical long audio base finished 1.5–2.3× sooner than tiny. The mechanism is not fully pinned down here (see METHOD.md).

Where these numbers come from

The measurements were taken on clipquill.com, a page that runs the model in the visitor's own tab and never uploads the file; the live timings in data/ are from that page over the public internet.

Contents

data/     the measurements, as CSV, plus a data dictionary
scripts/  the CDP harnesses that produced them, all runnable
samples/  reference transcripts for the eight WER clips
METHOD.md how each number was measured, and what is not covered

The headline table

Weighted WER by acoustic condition, 8 clips / 229 reference words:

conditionclipswordstinybase
clean synthetic TTS1336.1 %6.1 %
real speech, no noise22821.4 %10.7 %
real speech, light pink noise25754.4 %8.8 %
real speech, heavy pink noise28992.1 %33.7 %
real speech, telephone band + echo12263.6 %22.7 %
all822959.0 %19.7 %

base is better on 7 of 8 clips and tied on the eighth. It is not worse on any.

Running the scripts yourself

Everything is Node 22 with no dependencies (the CDP client is hand-rolled on the global WebSocket, and the page is driven over Runtime.evaluate). The two Python files are optional: serve-ab.py serves a build with the COOP/COEP headers the page needs, and cer-zh.py does the Mandarin normalizations.

node scripts/run-ab.mjs            # WER, both tiers, the eight clips
node scripts/run-timing.mjs        # same-material A/B timing
node scripts/run-timing-live.mjs   # cold/warm timing against the live page
node scripts/run-formats.mjs       # which containers decodeAudioData accepts
node scripts/run-zh.mjs            # the Mandarin clip
python scripts/cer-zh.py           # the three CER normalizations

The scripts contain absolute paths from the machine they were written on. Change the path constants at the top before running them elsewhere.

Two things that will bite you if you adapt these:

  • —Runtime.evaluate hangs while the page's main thread is busy loading or running the model. Poll with a short per-call timeout (8 s) and treat a timeout as "still busy", not as a failure.
  • —DOM.setFileInputFiles needs the backend node id, which is nested under node in the result of DOM.describeNode.

Reuse

Data and documentation are CC BY 4.0 (LICENSE). Scripts are MIT (scripts/LICENSE). If you reuse the data, please keep the attribution and say which measurement date you are citing — the numbers move when the model or the page changes.

Archived versions and DOI

This dataset is archived on Zenodo. Cite the version DOI for one specific release, or the concept DOI to point at the dataset as a whole.

  • —Concept DOI, all versions: https://doi.org/10.5281/zenodo.22826968
  • —Version DOI, v1.0.1: https://doi.org/10.5281/zenodo.22840611
  • —Version DOI, v1.0.0: https://doi.org/10.5281/zenodo.22826969

The same files are also published as a Hugging Face dataset: https://huggingface.co/datasets/sophia8888/clipquill-asr-benchmark

Citing

See CITATION.cff. Author is listed as Clipquill; if you are the owner and want your own name there, change it in CITATION.cff and codemeta.json before depositing.