Local vs Cloud LLMs: Tradeoffs Explained

Two paths for running an AI model: rent one in the cloud, or run your own on hardware you control. The advice online usually picks a side. That's backwards — the right choice depends on which constraint is actually binding for you. Here's the mechanism behind each tradeoff, walked through one real decision, and a rule you can apply to your own case today.

Maya manages the front office at a three-therapist physiotherapy clinic. She has a patient intake note open — history, medications, a paragraph about a workplace injury — and she wants a two-line summary to paste into the treatment plan. Her cursor is hovering over a cloud chatbot's text box. The note is in her clipboard. She stops.

Where, exactly, does this text go when she hits enter? Onto a server owned by a company she has a free account with, in a data centre she'll never see, governed by a policy she hasn't read. The patient signed a form about her clinic handling their records. They didn't sign anything about a third party.

That pause is the whole decision. "Local or cloud" sounds like a technical preference — a question about hardware and setup. It isn't. It's a question about which real-world constraint you're up against, and whether the obvious default happens to break it. If you're new to how these models work at all, what a large language model actually is is worth five minutes first; here we're past that, standing where Maya is, deciding where the model should live.

Two places a model can run

Strip away the jargon and there are two arrangements.

Cloud means the model runs on someone else's computers. ChatGPT, Claude, Gemini — you send your text over the internet, their hardware does the work, the answer comes back. You rent capability by the request.

Local means the model runs on your computer — a laptop, a workstation, a server in the office. Tools like Ollama let you download an open-weight model and run it with no internet connection at all. You own the capability outright.

Everything people argue about — privacy, cost, speed, quality, upkeep — flows from that one difference in where the computation happens. Walk each one as a mechanism, not a slogan, and the choice stops being a matter of taste.

Privacy: a boundary, not a promise

Start where Maya started, because for her this is the dimension that overrules the rest.

When she pastes the note into a cloud service, the text physically leaves her network. What protects it after that is a contract — a privacy policy, maybe an enterprise data-handling agreement. That can be a perfectly good protection. But notice its shape: it's a promise from someone else that they'll behave. To trust it, she'd have to verify their controls, their sub-processors, their retention, and defend all of it to an auditor who asks a simple question: can you prove this record never left your custody?

When the model runs locally, the answer to that question is structural. The text never crosses the network boundary, so there's nothing to promise and nothing to audit off-site. "We contractually agree not to misuse your data" and "the data physically never left the building" are different categories of assurance. For a regulated record — patient history, privileged legal communication, sensitive financials — the second category is the one compliance teams can actually stand behind.

That's the mechanism: cloud converts a data-custody question into a trust-and-verify problem; local removes the question by never moving the data. When a rule or contract says the data cannot leave your control, privacy isn't a tradeoff you weigh against convenience — it's a gate. Local is the only side of it.

Cost: a bill that scales with use, versus one you pay once

Now suppose Maya's data weren't sensitive. Would cost push her either way?

Cloud pricing works like a metered utility: you pay per unit of text in and out. At low volume this is almost free — a few summaries a day is pocket change, and you'd be foolish to buy hardware to avoid it. The mechanism that bites is linearity. The bill grows in lock-step with usage, and usage has a way of quietly growing. One summary becomes a feature every therapist uses forty times a day; a document tool starts making several model calls per page across hundreds of pages. Nothing dramatic happens — the number just climbs, because you're renting, and rent is forever.

Local flips the curve. You pay up front for hardware (or use a machine you already own), and after that each request costs essentially nothing but electricity. High, steady volume is where that pays off: a one-time cost amortised across a flood of requests. The catch is the mirror image of cloud's — you've spent the money whether you use the model once or a million times, and you now own a box that needs looking after.

So the honest comparison isn't "which is cheaper." It's cheaper at what volume, for what kind of task. Light or spiky use rewards renting. Heavy, predictable use on tasks a modest model can handle rewards owning. Precise break-even numbers depend entirely on your prices and volumes, so treat any specific figure as illustrative, not a promise — the shape of the two curves is the durable part.

Speed and offline: where the request has to travel

Cloud adds a round trip: your text goes out over the network and the answer comes back. For a single question that's usually imperceptible — the providers run fast hardware. But it's a hard dependency on a working connection and their service being up. No internet, no answer. Their outage is your outage.

Local answers without leaving the room, so it works on a plane, in a clinic with flaky wifi, or during someone else's outage. Whether it's faster depends on your hardware: a small model on a decent GPU can feel instant, while a large model on a modest machine can be slower than the cloud. The trade is control over the dependency, not a guaranteed speed win — you've swapped "depends on the provider" for "depends on your own machine."

Capability: the frontier still lives in the cloud

Here honesty matters, because it's where beginners are most often misled in both directions.

The most capable models available today are cloud models. They're trained at a scale — money, data, hardware — that you cannot reproduce on a workstation, and for genuinely hard work (subtle reasoning, long documents, multi-step problems) the gap is real and measurable, not marketing. If the task is hard, the cloud frontier is still the sharper tool.

But most everyday tasks are not hard. Summarising a paragraph, sorting tickets into five buckets, extracting fields from a form, drafting routine text — a well-chosen local model does these perfectly well. Reaching for the frontier here is like hiring a specialist surgeon to apply a plaster. The skill you understand about a model matters more than raw power: knowing what LLMs can't reliably do keeps you from trusting either a cloud or a local model with a job it will quietly get wrong. Capability isn't "which model is smartest." It's "does this task need the frontier, or is it routine?"

Upkeep: convenience you rent, or control you maintain

The last dimension is the one that's easy to forget until it's Tuesday and something broke.

Cloud hands you the operational work as a package. Sign up, get a key, make requests — no drivers, no model files, no updates. In return you accept that the provider can change prices, retire a model, tighten a rate limit, or adjust a content policy, and you adapt on their schedule.

Local hands you control and the responsibilities that come with it. You pick and update the model, manage the hardware, and fix it yourself when it stops. Nobody can change the terms under you — and nobody's coming to help at 9pm either. It's the difference between renting an apartment and owning a house: the landlord can raise the rent, but the landlord also fixes the boiler.

The comparison, in one view

Dimension	Cloud	Local	Which wins when
Privacy	Data leaves your network; protected by contract	Data never leaves your machine	Local whenever a rule or contract forbids the data leaving
Cost	Cheap at low volume; scales up with every request	One-time hardware cost, then near-free per request	Cloud for light/spiky use; local for heavy, steady, simple work
Speed / offline	Fast per query; needs a live connection	No network trip; speed depends on your hardware	Local when you need offline or independence from a provider
Capability	Access to the strongest frontier models	Good and improving; a tier below the top	Cloud for genuinely hard tasks; local is fine for routine ones
Upkeep	Provider runs everything; they set the terms	You run and maintain it; you set the terms	Cloud to avoid ops; local when you must control the whole stack

No column wins every row. That's not a cop-out — it's the reason a single blanket recommendation is always wrong for somebody.

Back to Maya — walking it to an actual choice

(Developed example — composite scenario.)

Run Maya's situation down the table.

Privacy hits first and hardest. The intake note is a protected health record; her clinic's agreement with the patient doesn't cover shipping it to a third party. That's not a factor to weigh — it's a gate, and it's shut. The moment that's true, most of the other rows stop mattering.

But she checks them anyway, because they tell her whether local is practical, not just required. Capability: her task is a two-line summary of a short note — routine, not frontier work, well within a small local model's reach. Cost: three therapists generating summaries all day is steady, repeating volume — exactly the shape that rewards a one-time hardware cost over a forever-growing bill. Upkeep: this is the real price. Someone at the clinic now owns a machine, a model, and the updates. Maya isn't technical, so "someone" means a few hours of setup help and a plan for who to call when it misbehaves.

Her decision isn't "local is better." It's specific: the privacy gate forces local, and capability and cost confirm local is workable, so the only open question is who owns the upkeep — and she books an afternoon with a contractor to stand up Ollama on the office workstation rather than pasting one more note into a browser.

Notice what didn't happen. She didn't pick a side from a blog and rationalise it. She let the binding constraint pick, then checked the rest for practicality. Change one fact — make the note non-sensitive — and the gate opens, capability and cost both point to the trivially easy option, and cloud wins the same afternoon.

The decision rule you can run today

You don't need to memorise five arguments. You need to ask them in the right order, because the first "yes" usually ends the conversation.

1. Does a law, contract, or policy forbid this data from leaving your control? If yes, run local — privacy is a gate, and you're done. If no, keep going.

2. Is your task genuinely hard — deep reasoning, long documents, multi-step work? If yes, default to cloud; that's where the frontier lives. If it's routine, keep going.

3. Is your volume high and steady, and growing? If yes, price out local — do the math on hardware-once versus bill-forever, and factor in who maintains it. If your use is light or spiky, stay on cloud; you'll spend less and babysit nothing.

Most beginners land on cloud at step 2 or 3, and that's the correct starting place: best models, zero setup, tiny cost until you're doing real volume. Move to local when a specific answer above sends you there — a privacy gate, or a volume-plus-simplicity case that pencils out — not because local sounds more serious. Plenty of teams end up running both: cloud for the hard jobs, local for the routine, high-volume, or sensitive ones. That's not indecision; it's matching the tool to the constraint, which is the entire skill.

If you're still building your feet under the whole topic, a practical guide to what AI is sets the ground under this decision. But the rule above is the part to keep: don't ask which is better. Ask which constraint is binding — data, difficulty, or volume — and let it point. Run those three questions against the very next task you were about to hand a model, and you'll know, in under a minute, where that task should live.