passpoo/talyno-whisper-small-finance

talyno-whisper-small-finance

A LoRA fine-tune of openai/whisper-small, specialized for earnings-call and investor-meeting speech recognition in English and Chinese. This is the merged PyTorch/safetensors checkpoint used as the conversion source for `passpoo/talyno-whisperkit-coreml`, the WhisperKit CoreML build shipped in the Talyno iOS app.

• —Base model: openai/whisper-small (244M params)
• —Fine-tuning method: LoRA (PEFT), encoder frozen, adapters on q_proj / v_proj
• —Languages: English (en), Chinese — Simplified + Traditional (zh)
• —Format: PyTorch / safetensors (merged adapter, not a bare PEFT checkpoint)
• —License: Apache 2.0

Training data & methodology

Two sequential training phases on the same LoRA adapter, merged into the base model afterward:

Key details:

• —The encoder is frozen throughout training — only the decoder's attention projections receive LoRA adapters, preserving Whisper's general acoustic representations while adapting decoding to financial vocabulary and IR/analyst-Q&A phrasing (EPS, EBITDA, guidance, etc.).
• —Chinese training data is a mix of real speech (FLEURS) for acoustic diversity and TTS-synthesised sentences to guarantee coverage of financial terms that are rare in general speech corpora. Traditional-Chinese vocabulary coverage in particular comes from the TTS set, since FLEURS has no Taiwan Mandarin (cmn_hant_tw) locale.
• —Before merging (merge_and_unload()), generation_config.forced_decoder_ids and config.forced_decoder_ids are explicitly cleared. Whisper's set_prefix_tokens(language=...) calls during training otherwise leave a language token baked into the merged model's generation config, which forces every inference to one language regardless of input.

Downstream conversion

This checkpoint is the conversion source for the WhisperKit CoreML build hosted at `passpoo/talyno-whisperkit-coreml`, using whisperkittools. Note: conversion currently requires pinning transformers==4.42.4 in the conversion environment — newer transformers versions changed past_key_values handling in a way that's incompatible with the whisperkittools decoder tracer (see the finetune guide for details).

Datasets used (licensing)

• —SPGISpeech — earnings-call transcription corpus; see dataset card for license terms.
• —Google FLEURS cmn_hans_cn — CC-BY licensed.
• —TTS-synthesised financial sentences (proprietary sentence set, synthesized via Google Cloud Text-to-Speech) — not derived from any third-party corpus.

Limitations

• —English and Chinese only. Earlier development generations of this project experimented with a broader 6-language scope (adding German, Spanish, Japanese, Korean); the model actually published here was retrained and narrowed to EN + ZH for training-data quality and maintainability reasons.
• —Optimized for earnings-call / investor-meeting register — not benchmarked on casual conversation or other domains.

License

Apache 2.0, consistent with the base openai/whisper-small model.