Running Whisper locally: speech recognition without the cloud

AI speech recognition without the cloud, everything on your own machine: at first that sounds like a contradiction. Speech recognition always used to mean a big server somewhere that you send your voice to. That has not been true for a long time. I built exactly this technology into a dictation tool of my own, and in doing so I learned what really works locally and where the marketing promises stop. Straight talk, no hype.

What Whisper actually is

Whisper is a speech-recognition model that OpenAI made freely available. Freely means it literally: you can download it and run it on your own machine, with no account, no subscription and without a single recording leaving the device. It is not the only local model, but it is by far the most widely used. A whole ecosystem of lightweight runtimes has grown up around Whisper, and that is what makes it practical to run on ordinary office machines in the first place.

An important point for context: Whisper is a tool, not a finished product. You need something that launches the model, drives the microphone and writes the text somewhere. That is precisely where the finished dictation apps differ from one another. The model underneath is often the same one.

What really works locally

Let us start with the good news, because there is more of it than many people expect. A decent local setup transcribes dictation today with surprising cleanliness, offline, with no internet and no wait for a server. For running text, file notes, findings and meeting minutes, the quality is more than sufficient as a rule.

• No data outflow: the recording is processed on your machine and is then either gone or stays with you as a file. No third-party servers, no third country, no need for a legal transfer basis. For confidential professions this is the decisive point.
• No recurring costs: no minute quota, no subscription. Once set up, every further hour of dictation costs nothing but electricity.
• Independence: no provider that can switch off the service, raise the price or change the terms. The model will still run in five years exactly as it does today.

The underlying logic is simple: what never leaves the machine, I do not have to secure by contract. For law firms, medical practices and tax advisers this is more than convenience; it is the simplest answer to a professional duty of confidentiality.

The model question: small and fast versus large and accurate

Whisper comes in several sizes, from tiny to large. And here lies the first honest truth: you cannot have both. A small model runs on almost anything and delivers instantly, but it makes more mistakes, especially with names, technical terms and numbers. A large model is considerably more accurate, but it wants more computing power and takes longer.

For most office use, a fast variant of the large model has proven to be a good middle ground: close to the accuracy of the largest models, but noticeably quicker. Add to that fine-tuned variants for a specific language, which do even better with that language's diacritics, compound words and typical technical terms. Given the choice, you take a model tuned for your working language rather than the plain English default.

The question is never which model is the best. The question is which model, on your hardware and in your time budget, delivers the accuracy your texts require.

The hardware truth

Now the part the glossy demos like to leave out: locally, the hardware is not free. You do not need a supercomputer, but nor will the ten-year-old office PC do, the one that already breaks a sweat opening a PDF.

• Apple Silicon Macs (M1 and newer) are surprisingly strong for local speech recognition, because the model can use the built-in graphics and neural units.
• Windows and Linux machines with a dedicated graphics card play in the same league; the graphics card takes on the heavy lifting.
• A CPU alone, with no graphics card, also works, above all with the smaller and the fast models, just more slowly. For dictation that you are going to edit anyway, that is often perfectly adequate.

As a rule of thumb: a machine that is up to current office work can handle local dictation. If it also has a reasonably modern graphics unit, it runs smoothly rather than patiently.

Where local runs into limits

So that this does not turn into an advertisement: there are things that pure local speech recognition cannot do, or cannot do on its own. Anyone expecting them will be disappointed, and that is not a flaw in the model but a mistaken expectation.

• Who is speaking when: cleanly attributing a conversation with several speakers, that is, speaker separation (diarisation), is not something Whisper does out of the box. It is possible, but it needs additional building blocks.
• Perfect formatting: paragraphs, headings and style templates do not appear by themselves. You get clean running text with punctuation; the finishing touches are done by you or by a downstream stage.
• Specialist vocabulary and proper names: special terms, file references and product names are not always right the first time. Good tools allow custom word lists; the model on its own is guessing.
• Very poor recordings: heavy reverberation, several voices talking over each other, building-site noise. What a human can barely make out, the machine cannot either.

None of this is a deal-breaker. But it is the difference between a realistic view and a demo promise.

When local wins, when the cloud does

In the end it is a trade-off, not a matter of faith. Local wins clearly when confidentiality matters, when a lot is dictated and the running costs hurt, or when you want to stay independent of a provider. For law firms, medical practices and tax advisers, usually all three apply.

The cloud keeps its place when the hardware really is weak, when you need convenience and collaboration features that go beyond plain dictation, or when you simply have no appetite for a one-off setup effort. Both are legitimate. It should just be a conscious decision, and not one that the most convenient default makes for you.

The honest closing line

Local speech recognition is no longer a niche tinkering topic. It is mature enough for everyday use, provided you know which model and which hardware fit together and where the limits lie. For everyone who professionally holds other people's secrets, it is the calmest route: what never leaves the machine cannot end up with someone who has no business with it. No wizardry. Just a conscious decision.