Choosing a Whisper model size
Why a bigger speech model is not simply better, what changes between tiny, base, small and large-v3-turbo, and the failure mode that made us ship a 547 MB model.
The obvious way to pick a speech model is to sort the available sizes by accuracy, then take the largest one that still runs fast enough. That heuristic happens to land on the right answer for Voicecape, and it lands there for the wrong reason — which matters, because the wrong reason stops working the moment a new language is added.
The decision is also not reversible in the usual way. The model ships inside the app bundle rather than downloading on first run, so there is no screen where someone trades size for accuracy after the fact. One file is chosen once and every installation carries it, in airplane mode or on a fresh Mac with no network at all.
Everything below comes from two files in the repository: the model fetch script, which carries the comparison table, and a benchmark record from an M1 Max that states its own limits — a single machine, recognition time only. Those limits are worth keeping in view while reading any number here.
The sizes and what actually changes between them
The whisper family runs roughly tiny, base, small, medium and large, with large-v3-turbo as a large-family model built for faster decoding. Each step multiplies the parameter count, and the improvement is not spread evenly. Common English words in clean audio are close to solved at every size; what moves as the model grows is the hard tail of proper nouns, domain vocabulary, and any language that is not English.
Measured on an 11-second clip on an M1 Max, with whisper.cpp at commit 080bbbe8:
- tiny.en-q5_1 — 31 MB, 188 ms, real-time factor 0.017
- base-q5_1 — 57 MB, 222 ms, real-time factor 0.020
- small-q5_1 — 181 MB, 464 ms, real-time factor 0.042
Real-time factor is processing time divided by audio length, so 0.042 means an 11-second clip took under half a second to decode. Lower is faster.
The failure that ruled out base
On the same test clip, base rendered the Korean noun 견적서 — a price quotation, the kind of word that appears in the first sentence of a work message — as 현척서, which is not a word at all. small produced 견적서 correctly. That alone would make base the weaker choice, but it would not necessarily make it disqualifying; a visibly wrong noun is something you notice and fix.
The second failure is the one that decided it. base also collapsed a repeated word: 그 그 came back as 그. The recogniser deleted a token the speaker actually said, before any of the product’s own code ran.
That breaks a promise structurally rather than occasionally. Verbatim mode exists to hand back what was said, including the hesitations, and it cannot preserve something the recogniser has already removed. Exact Clean has the mirror problem: it can only take out what it was given, and every rule it applies assumes the transcript still contains the speaker’s actual tokens. A recogniser that silently deduplicates turns the difference between the two modes into a claim nothing downstream can keep.
What quantisation buys and what it costs
Every candidate in that table is a quantised build, which the file names announce — q5_1 for the small models, q5_0 for the shipped one. Quantisation stores the model’s weights at reduced precision rather than at full floating point. The file gets substantially smaller, less data has to move between memory and the compute units for each token, and both effects are why a half-gigabyte file can sit inside an app bundle at all.
What it costs is accuracy, in amounts that vary by model and by language, and this is the point where honesty beats a confident number. Every comparison recorded here is quantised against quantised. No measurement of the same model at full precision exists in the repository, so there is no figure to quote for what quantisation itself took away — only for the differences between the quantised builds, which were large enough to change the decision.
The languages that raised the floor
small was the shipped model for a while and the reasoning above looked settled. Then Spanish, Japanese and Chinese were added, the same test was re-run against large-v3-turbo-q5_0, and small failed all three in three different ways.
In Spanish, "Todavía no he recibido el presupuesto" came back as "To Davia Noje Recibido El Presio Pusto". That is not an accent problem or a word or two going astray; the recogniser stopped producing Spanish words. In Japanese, 見積書 became 三森省 and ご返信 became ご変身 — the same shape as the Korean failure that disqualified base, a business noun destroyed while the sentence around it looks fine.
The Chinese failure is a different category again. Simplified input came back in Traditional characters, three times out of three. The content was right every time and the script was wrong, which no cleanup rule can repair — cleanup removes speech noise, it does not convert between writing systems — and which a reader who works in Simplified simply cannot use.
large-v3-turbo-q5_0, at 547 MB, got all three right. For English and Korean its output was byte-identical to small’s. The upgrade bought nothing at all for the two languages that were already working, which is exactly the shape of this kind of decision: the model size is set by your worst language, not your average one.
Speed as the axis that stopped mattering
On Apple silicon, the constraint people expect to bind does not. The 11-second clip decodes in 464 ms with small, and the earlier benchmark record — base.en, Metal, M1 Max, macOS 15.7.4 — logs 529.25 ms total including 89.91 ms of model load, a real-time factor of 0.048, roughly twenty times faster than the speech itself.
Going from small to large-v3-turbo, a file three times the size, took 19.4 seconds of speech from 0.67 s to 1.22 s. That is around sixteen times real time. The step that quadrupled the bundle cost about half a second on a twenty-second dictation, and quality kept improving long after speed had stopped being the thing standing in the way.
Those figures deserve their caveats stated rather than buried. They come from one machine, an M1 Max, and the benchmark file says in its own text that a top-specification number must not be published as the representative one. They also measure recognition alone, not the recording, cleanup and insertion steps that make up what a person actually waits for. The accurate summary is therefore not a speed claim: speed was not the deciding axis, and accuracy on the worst-performing language was.
The bill for the decision
The cost is real and it is paid by everyone: the shipped model added 366 MB over small, and the download is correspondingly large for a utility that inserts text at a cursor. There is no honest way to present that as anything other than the price of the choice.
What it buys is the absence of a step. Nothing is fetched on first run, there is no per-language download when you switch, and the app works with the network off from the moment it is installed. The size is a one-time install cost; a download standing between "I installed it" and "I said something" is a recurring one, paid by every new person, and it is the kind of step people abandon. The alternative to carrying the file is not a smaller download but a different privacy model.
One more consequence falls out of the multilingual choice: whisper has no per-language models outside the English-only variants, so a single file covers all five languages, and there is nothing to swap when you change language mid-sentence.
How to read a model-size recommendation
The general advice — bigger is better, subject to speed — is not wrong so much as it measures the wrong thing. Size is set by the worst case rather than the average, so test with the vocabulary you actually dictate rather than with read-aloud prose: the invoice nouns, the product names, the surnames of people you email. Then rank the failures by whether you would notice them, because a mangled word costs one correction and a dropped word costs something you may never find. That is why the Korean measurement decided this rather than the English one, even though English is the language most of the testing was done in.