Modelpublic

Numbat/skyrim-whisper-base.en

sourceHugging Facemitupdated 2y agoView on Hugging Face

0likes3downloads

Model Card

license: mit language:

—en base_model:
—openai/whisper-base.en ---

Whisper-base.en Finetune for Skyrim Terminology

This is a finetune of whisper-base.en that improves transcription of fantasy names and terms from Skyrim while maintaining general English accuracy.

Use as a drop-in replacement for https://huggingface.co/openai/whisper-base.en

Performance

—Common Voice 11 test set WER: 27.17 (Base model: 26.99)
—Skyrim-specific terms error rate: 20.49% (Base model: 50.82%)

Training Data

—5,344 Skyrim voicelines from the Game (average ~7s each)
—Voicelines selected to contain target fantasy words/phrases

Finetuning Approach

Initial attempts decreased general English performance. Solution: freezing 3 decoder layers plus the encoder maintained Common Voice accuracy while significantly improving Skyrim term transcription.

Cherry-Picked Results from Eval

Source	Line
ground truth	`Falkreath`'s banner bears the stag. They say it's because they'll never let a woman rule the hold.
this model	`Falkreath`'s banner bears the stag. They say it's because they'll never let a woman rule the hold.
whisper-base.en	`Fulkreet`'s banner bears the stag. They say it's because they'll never let a woman rule the hold.
ground truth	Her daughter was sent to `Whiterun`, the skirmishes there have been violent. The `whiterun` `legate`, he needed to know the `Stormcloak` positions.
this model	`Hadata` was sent to `Whiterun`. The skirmishes there have been violent. The `Whiterun` `Legate`, he needed to know the `Stormcloak` positions.
whisper-base.en	Her daughter was sent to `White Run`. The skirmishes there have been violent. The `White Run` `laggot`, he needed to know the `storm clock` positions.
ground truth	I never heard of `Tiber Septim` killing any dragons.
this model	I never heard of `Tiber Septim` killing any dragons.
whisper-base.en	I never heard of `Tiberceptom` killing any dragons.
ground truth	Drink `Honningbrew Mead` with a `wench` on each arm? Ha ha!
this model	Dring `Honningbrew mead` with a `wench` on each arm?
whisper-base.en	During `Honning Broomade` with a `winch` on each arm?
ground truth	`Riften, eh`? Probably down in the `Ratway`, then. It's where I'd go.
this model	`Riften-Ei`? Probably down in the `Ratway` then. It's where I'd go.
whisper-base.en	`Rift in A`. Probably down in the `rat way` then. It's where I'd go.
ground truth	`Volunruud`? Well, that is interesting. I know this place...
this model	`Volenrud`? Well, that is interesting. I know this place.
whisper-base.en	`Volon Rood`? Well, that is interesting. I know this place.
ground truth	He proposed that the real reason `Saarthal` was fought over by the Elves and the Nords was because something was unearthed there.
this model	He proposed that the real reason `Sauthal` was fought over by the elves and the Nords was because something was unearthed there.
whisper-base.en	He proposed that the real reason `Sothaw` was fought over by the elves and the Nords was because something was unearthed there.
ground truth	Yes, but a `skeleton` has many bones, and they all look alike. A bone was stolen and switched with a `nameless one`, by those loyal to `Ysgramor`'s kin.
this model	Yes, but `Eskeletan` has many bones, and they all look alike. A bone was stolen, switched with a `Nameless One` by those loyal to `Ysgramor`'s kin.
whisper-base.en	Yes, but a `skeleton` has many bones, and they all look alike. A bone was stolen, switched with a nameless one by those loyal to `Isgrimor`'s kin.
ground truth	Then you'll need to find `Elenwen`'s office and search her files. `Malborn` should be able to point you in the right direction.
this model	Then you'll need to find `Elenwen`'s office and search her files. `Malborn` should be able to point you in the right direction.
whisper-base.en	Then you'll need to find `Ellenwyn`'s office and search her files. `Malborn` should be able to point you in the right direction.
ground truth	`Pa` was a sailor with a kid in every port from `Seyda Neen to Stros M'Kai`. One of them ports is `Windhelm`, and one of them kids is me.
this model	`Paar`'s a sailor with a kid in every putt from `Say the Name, Dostros Macai`. One of them ports is `Windhelm`, and one of them kids is me.
whisper-base.en	But I was a sailor with a kid in every port from `Saturday in Dosto, Macai`. One of them ports is `wind-hulme`. And one of them kids is me.
ground truth	At least I have an excuse. I'm obligated to love my sister. `Leigelf` chose to.
this model	At least I have an excuse. I'm obligated to love my sister. `Legelf` chose too.
whisper-base.en	At least I have an excuse. I'm obligated to love my sister. The `gels` chose too.
ground truth	`Izarn` says this place is no different than `Cidhna Mine`. The fool can say that because he's never been there.
this model	`Isan` says this place is no different than `Cidna Mine`. The fool can say that because he's never been there.
whisper-base.en	`Eason` says this place is no different than `Sid and mine`. The fool can say that because he's never been there.
ground truth	`Boethiah`, `Mephala`, and `Azura`. When I sought to steal their relic, they chose my own `pupil` as their retribution.
this model	`Boethiah`, `Mephal` and the `Zul`. When I sought to steal their relic, they chose my own `pukule` as their retribution.
whisper-base.en	`Boethi`, `Mephal` and the `Zula`. When I sought to steal their relic, they chose my own `pupil` as their retribution.
ground truth	`Vilkas`, take him out to the yard and see what he can do.
this model	`Vilkas`, take him out to the yard and see what he can do.
whisper-base.en	`Vilcus`, take him out to the yard and see what he can do.