On-Device or Cloud: Where Should the AI Run?

Press a shortcut, talk for twenty seconds, and watch the words appear on screen. The audio never leaves the machine. Then an optional second step strips the ums and the false starts, and that step calls a model running in a data center somewhere else entirely.

That is one small feature answering the on-device question twice, in opposite directions, about ten seconds apart. Most AI features end up in the same place. The useful question is not "local or cloud." It is "which part of this job goes where."

Six questions that settle it

When we are deciding where a piece of inference runs, we work through the same short list. The answers rarely all point the same direction, and that is the point. You are looking for the seam, not the winner.

• Data sensitivity: What exactly leaves the machine, and would you be comfortable writing that sentence on your product page? "Your voice never leaves your Mac" is a claim worth engineering for. "We send your audio to a vendor who says they do not train on it" is a different claim, and some people will not accept it at any price.
• Latency: Not average latency. The latency of the slowest step in the loop the user is actually waiting through. A network round trip is often smaller than you fear and occasionally ruinous, because it is the variance rather than the mean that people notice.
• Cost shape: A hosted model costs a little every time. A local model costs a one-time download and then nothing. A feature someone uses fifty times a day has a very different profile from one they use twice a month.
• Offline: Does this have to work on a plane, in a basement, on conference wifi that resolves DNS and nothing else? If the honest answer is yes, the cloud is not available for that path. It is only available for the enhancement layered on top of it.
• Capability ceiling: Is there a small model that is genuinely good enough at this job, or are you hoping one will be? Those are different situations, and you can tell them apart by testing.
• Battery and thermal: Local inference is not free. It is paid in watts and fan noise instead of dollars, and on a laptop that is a real budget with a user-visible limit.

Speech to text has crossed over

Transcription is the clearest example of a job that used to require a server and no longer does. Open-weight speech models run comfortably on a current laptop, in many languages, fast enough to put words on screen while you are still talking. In Voice, our Mac dictation tool, Whisper runs on your Mac and the audio is never uploaded. After a one-time model download, that path keeps working with the network unplugged.

There is a size ladder here and it is worth walking. The default is a multilingual model that is a few hundred megabytes on disk. There are lighter English-only models that load faster and use less memory and are perfectly adequate for clear speech in a quiet room. macOS also ships its own dictation, which costs nothing to download at all. Which rung is right depends on whether your users are transcribing careful desk speech or mumbling in a car, and you find that out by trying it rather than by reading about it.

What makes transcription tractable locally is that it is a narrow, well-defined mapping. Audio in, text out, no world knowledge required. Jobs with that shape tend to be where small models catch up first.

Cleanup has not

The second half of that same feature is a different animal. Turning "so um I wanted to let you know that the the design review moved to Thursday at three" into something you would actually send requires deciding which repetitions were stutters and which were emphasis, whether a spoken number should be a numeral, that "new paragraph" was an instruction and not a phrase to transcribe, and where the sentence boundaries belong in speech that had none.

That is judgment about language, and judgment is exactly what the capability ceiling question is asking about. A frontier model does it well. A small local model does it inconsistently, and inconsistently is the worst possible failure mode for a tool someone is about to paste from, because it means they have to check every single time. A tool you have to check is not saving anyone anything.

So that step goes to a hosted model, and the important detail is what gets sent: the text transcript, not the audio, and only when the user has turned the step on. The sensitive artifact stayed home. The part that needed a bigger brain went out.

The cost shapes differ, not just the sizes

Resist comparing a per-call price to a per-call price. Compare shapes.

A local model is a fixed, visible, one-time cost: a download the user waits for once, disk space they gave up, memory pressure while it runs, and battery for every invocation. A hosted model is a small recurring cost that scales with use, plus a dependency on someone else's uptime and someone else's pricing decisions, plus a credential that has to live somewhere.

The more interesting consequence is that they fail differently. Local inference degrades. It gets slower under thermal pressure, it competes with whatever else the user is doing, and it produces weaker results on a weaker device. Hosted inference goes down. It behaves exactly as expected until it does not work at all. If your feature needs to survive a provider outage, that is an architecture constraint rather than an ops problem, and it points at a local fallback you have to actually build and actually test.

What the split buys you

Splitting the pipeline is not a compromise between two worse options. It produces things neither pure approach gives you.

You get an honest privacy claim about the sensitive part, which is almost always the raw capture: the audio, the frames, the screen. Raw capture is where the incidental information lives, the background conversation and the unrelated browser tab, and it is the part users are right to be protective of. Text is easier to reason about, easier to show someone before it goes anywhere, and easier to redact.

You get a fallback nearly for free. If the cloud step fails, you still have a transcript. The feature is worse, not broken, and that is a much better place to stand than a spinner and an error.

You also get an upgrade path. Local models keep improving. If a small model becomes good enough at cleanup, the boundary moves and the hosted call disappears. That only works if the boundary is a seam in your code rather than an assumption threaded through it, which is a good argument for keeping the model behind a setting instead of a constant.

Where we land

Default to local for capture and for narrow transformations with a clear right answer. Default to hosted for anything that needs judgment, world knowledge, or a lot of context held at once. Put the boundary at the point where the data becomes less sensitive and more structured, which is usually the moment raw input becomes text.

Then go verify the claim you are making. If your page says the audio stays on the device, someone should be able to pull the network cable and watch it still work. If it does not, the architecture is wrong no matter what the diagram says.

If you are weighing this for something you are building, tell us what it does and we will tell you where we would draw the line.