On-device AI vs cloud AI is no longer a single choice made for a whole product. Current phones run small models locally and send harder requests to large cloud models, so the practical question is now which feature runs where, and what evidence supports that choice.
What changed: phones now run AI locally and route between device and cloud
A TianPan blog post describes Apple routing requests between on-device models and its Private Cloud Compute servers, and Google's Gemini Nano escalating complex requests to cloud Gemini. According to a Kanopy Labs blog, Samsung's Galaxy AI uses on-device Gemini Nano with cloud fallback for translation and photo editing. These are reported patterns. We did not verify product launch dates, so confirm them before citing.
Local inference itself is not the new part. Edge inference is an established category that covers phones as well as sensors, cameras, vehicles and appliances, according to a GMI Cloud vendor blog. What appears new is hybrid routing as the default in consumer products, where the device handles most requests and the cloud handles the rest.
Where the model runs and what data moves
On-device inference runs a trained model on the hardware in front of you. Cloud inference sends your input to a remote server, which runs the model and returns a result. With on-device inference, the input for that request does not cross the network. With cloud inference, it does.
The division of labour follows from resources. An arXiv preprint notes that cloud systems have far larger resources and are the natural home for storage, data processing and model training, while devices are constrained by form factor, cost and energy.
Latency: the network is not the only variable
The trade-off is between two delays. Cloud inference is computationally fast but pays transmission time to reach the server. On-device inference avoids transmission but may compute more slowly on constrained hardware, according to an arXiv paper on computation and transmission latency.
Blog figures are often quoted here. Apptitude says a cloud API call adds roughly 100–500 ms of round-trip time, and that a quantized image classifier on an iPhone 15 Pro returns in 5–15 ms. Neither figure cites a measurement, so treat both as illustrative.
A simulation is more useful for the point that removing the network is not automatically faster. In one simulated autonomous-braking scenario in an arXiv preprint, the cloud configuration, with a median round trip of 22 ms, braked earlier and stopped farther from obstacles than the on-device setup. The cloud could run a larger, more accurate model on faster compute. This is one simulated scenario in a preprint, and it cannot be generalised.
Privacy depends on architecture, not only location
The core argument for on-device privacy is simple. The TinyML survey says on-device inference keeps data inside the device rather than on cloud servers. The general cloud concern, as an arXiv preprint puts it, is that transmitted data can be intercepted and stored indefinitely.
That is not the whole picture. A WaveSpeed practitioner guide argues that cloud privacy depends on architecture, and that cloud processing can be designed around stateless processing and restricted access. Apple's Private Cloud Compute is the most cited example. A Vucense summary describes its nodes as custom server hardware with the same security technologies as iPhone, stateless processing, and public transparency logs for software images. That is a secondary summary; we did not retrieve Apple's primary page.
Architecture does not cover every route. A SimpleMDM blog reports that data sent to third-party services such as OpenAI is not covered by Private Cloud Compute guarantees. That is one blog's reading, so check what each feature documents before assuming your data stays local.
Cost, battery and hardware limits
Cloud inference is billed per request. Once a model is downloaded, on-device inference carries no per-request charge, as Apptitude notes. Its worked example, which is hypothetical arithmetic and not a typical price, comes to about $150,000 a month for 50 inferences per user per day, 100,000 users and $0.001 per inference.
No per-request charge does not mean no cost. Battery drain, storage for multi-gigabyte models and update pipelines all carry costs that a cloud invoice hides. Memory is also fixed and shared by model weights, the context cache and the runtime, and a developer blog reports that a cold start can dominate first-response time.
NPUs are much more power-efficient than CPUs or GPUs for the same work, but they still use energy. High-duty local models can drain battery, spike RAM and trigger thermal throttling on mid- and low-tier hardware. Phones now run local models in roughly the low-single-digit billions of parameters, according to a Docam article, and sources disagree on exact sizes.
Where cloud still leads
Spheron, a cloud GPU vendor, says cloud GPUs are the only practical option for frontier-model inference. LLM Hub expects cloud models to stay ahead at the frontier while on-device models handle more everyday tasks. Both are opinions. No source we found measures how much a quantized on-device model degrades output in practice.
When to choose cloud, on-device or hybrid
This frame is our synthesis of the sources above, not a finding from any one of them. Choose on-device when the work is frequent, latency-sensitive, privacy-sensitive or must work offline, since cloud inference requires connectivity and on-device does not. Choose cloud when the task needs the largest models or the freshest data. Choose hybrid when both matter, which is the pattern the large platforms are reported to use.
Before you commit, measure your own model on your target device with network time included. The published millisecond figures and the cost example are not enough to decide on.