One model, five languages
How a single multilingual speech model handles English, Korean, Spanish, Japanese and Chinese — including a sentence that switches language halfway through.
Voicecape recognises English, Korean, Spanish, Japanese and Chinese, and it does so with one 547 MB file that ships inside the app. There is nothing to download when you switch languages, no per-language pack, and no moment where the app is capable of one language and not another.
That is less a piece of engineering than a property of the model family: whisper has no per-language models apart from the English-only variants, so a multilingual build covers everything it covers at once. The interesting part is what the app has to do around it, because a shared model raises a question a per-language design never asks — which language is this sentence?
The behaviour described here comes from the settings core, the model catalogue, and the recogniser wrapper on the macOS side.
The function that stopped branching
There is a function in the model catalogue whose job is to pick a model for a language. It ignores its argument and returns the bundled model, and the comment above it explains that this is the point rather than an oversight.
The earlier version did branch. English got an English-only model, everything else got the multilingual one, and it looked like a free optimisation. The trap is that whisper’s English-only models do not fail when handed Korean audio — they emit English-looking transliteration. There is no error, no empty result, nothing that a test asserting "we got text back" would catch. The app could have shipped listing Korean as a core language while being structurally unable to produce a single Hangul character.
A wrong answer in the shape of a right one is worse than a refusal, which is the same reason a recogniser that silently deletes a repeated word disqualified an otherwise faster model.
What a single shared model saves
The mixed case deserves more emphasis than it usually gets, because for a lot of people it is not an edge case at all — it is how they talk at work. Any design requiring the language to be declared before the sentence begins handles that badly by construction.
- No download when you switch — the language setting is a setting, not an installation.
- One sentence can contain two languages. A Korean sentence with an English product name in the middle, or a Spanish sentence naming an English-language tool, is handled by the same decode rather than by picking a winner first.
- The personal dictionary is not gated behind the choice. When the language is set to automatic, the hint handed to the decoder is the union of every language’s dictionary.
The trade you are accepting
Multilingual models are not free, and it is worth saying so plainly. A model of a given size shares its capacity across every language it knows, so the same size trained on one language alone will generally do that language better.
The measurements available here are narrower than that general rule and worth quoting exactly. The multilingual build and the English-only variant of the same size were the same size on disk and produced identical English output on the clips that were measured. In the later comparison across model sizes, English and Korean output was byte-identical between the two candidates. Those are results on specific clips on one machine, not a proof that the trade costs nothing.
The reason to accept the trade where it does cost something is the switching cost you avoid. Five per-language models would mean either five files in the bundle or a download step at the moment someone first speaks a second language — and each of those models would separately have to clear the bar set by that language’s worst failure. One file that is slightly less specialised, and always present, is the better shape for a tool you reach for mid-sentence.
Automatic as an instruction rather than a value
The supported settings are automatic, English, Korean, Spanish, Japanese and Chinese, and automatic is the default. Internally it is treated as a request to decide later rather than as a language, and that distinction has a bug behind it.
Cleanup rules are per language, and the engine was once handed the literal string for automatic. It did not recognise it, fell through to the English rule set, and a Korean sentence came back with nothing removed — a feature that looked broken rather than misconfigured. The recogniser now exposes the language it actually decoded in, and that resolved answer, never the request, is what cleanup receives.
How detection decides
That last step is hysteresis, and it exists for a specific person: someone dictating in Korean who says an English product name mid-sentence should not find the next sentence transcribed as English. Sticking with what was working is wrong less often than re-deciding from scratch on every utterance, and a user who cannot see why the language changed has no way to stop it happening again.
whisper’s own detect-language flag is deliberately not used. It decides silently inside the main decode call, which leaves nowhere to put a confidence threshold, nowhere to consult the previous answer, and no way to show a result. The app runs detection as its own step precisely so that the answer exists as a value it can display.
- The mel spectrogram is computed, which is the same work the decoder does internally, so detection costs one extra pass over the audio.
- whisper’s language detection runs over that spectrogram and returns a language with a probability.
- The probability has to reach 0.85 to be accepted.
- Below that threshold, the language that last settled confidently stands, and the decode proceeds in it.
Automatic detection without a visible answer is worse than asking outright, so the on-screen indicator shows the language actually used — and shows it the same way when the language was set explicitly.
When to set the language explicitly
An explicit setting is obeyed and detection is skipped entirely. Two situations make that the better choice: dictating in one language nearly all the time, where detection solves a problem you do not have, and short utterances such as a few words into a search box, which are the hardest case for any detector because there is less signal to be confident about.
The personal dictionary interacts with this. Dictionary entries are stored per language, so an explicit setting scopes the hint to the terms belonging to that language. Under automatic, the app cannot know which language applies before you speak, so it uses every language’s entries together. The reasoning is that the dictionary is a hint rather than a constraint, so extra terms do no harm, while picking one language’s list would make the dictionary vanish the moment you spoke the other. Names remain the hardest category regardless of language, which is a problem of expectation rather than of audio.
What the shared model is not allowed to do
whisper can translate as well as transcribe, and translation is switched off. A Spanish sentence comes out in Spanish. Cleanup cannot reintroduce it either, since it is a rule-based stage that removes speech noise and has no capacity to produce a word you did not say — the boundary described in what the cleanup stage actually removes.
Carry-over between utterances is also off. Given the previous transcript as context, whisper will happily invent a continuation of a sentence you finished a minute ago, and inventing words is the one thing this product promises never to do. The dictionary hint is a different mechanism: a fixed vocabulary list, not the last thing you said.
One model, five rule sets
Sharing the model does not mean sharing everything downstream. Cleanup keeps a separate rule set per language, and those sets differ in kind rather than only in content. Japanese and Chinese have no spaces between words, so their filler removal works on substrings — a genuinely dangerous operation, deliberately confined to those two languages. Spanish rides the same whitespace-token path English does, with its own conservative list: bueno, pues, vale and este are all excluded, the last after a measurement in which removing the filler use of este also deleted the demonstrative in "este documento".
Korean sits between the two, with spaces that mark phrases rather than words, which changes what a filler rule can safely match. That is its own set of decisions, and the most interesting of them are about the rules that were left out.