The default assumption for most app teams is straightforward: call a cloud API, get a result, display it. For many AI features, that is the right call. But for a growing category of mobile use cases, running inference directly on the device produces a faster, cheaper, and more private experience. The tradeoff is not obvious, and getting it wrong costs you either performance or money. Here is how to think through the decision.
The Core Tradeoffs
Four factors drive the on-device versus cloud decision: latency, privacy, cost, and offline capability. Each one pushes in a different direction depending on your use case.
Latency is where on-device inference wins decisively. A cloud API call involves network round-trip time — typically 100-500ms just for the network hop, plus inference time on the server. On-device inference eliminates the network entirely. A quantized image classification model on an iPhone 15 Pro returns results in 5-15ms. For real-time features like camera-based object detection, AR overlays, or live text analysis, that difference is the gap between usable and unusable.
Privacy matters more than most teams initially estimate. On-device inference means user data never leaves the phone. For health data, financial information, biometric processing, or anything subject to HIPAA, GDPR, or SOC 2 requirements, keeping inference local simplifies your compliance story dramatically. No data transmission means no data breach vector for that feature.
Cost favors on-device at scale. Cloud inference charges per request. On-device inference costs nothing per request after the initial model download. If your app runs 50 inferences per user per day across 100,000 users, the cloud bill adds up fast — even at $0.001 per inference, that is $150,000 per month. On-device, it is zero marginal cost.
Offline capability is binary: cloud inference requires connectivity, on-device does not. For apps used in areas with unreliable connectivity — field service, outdoor recreation, international travel — on-device is often a hard requirement.
When On-Device Is the Right Choice
Certain use cases map cleanly to on-device inference because they demand low latency, involve sensitive data, or run at high frequency.
Image classification and object detection run exceptionally well on-device. Apple's Core ML can execute MobileNetV3 in under 10ms on recent iPhones. TensorFlow Lite achieves similar performance on flagship Android devices. If your app identifies plants, reads receipts, detects defects in manufacturing photos, or applies AR effects, on-device is the default choice.
Keyboard and text prediction has always been on-device for good reason — latency requirements are sub-50ms, the input is highly personal, and inference frequency is extremely high (every keystroke). Custom autocomplete or domain-specific text suggestion models belong on the device.
Real-time audio processing for features like noise cancellation, voice activity detection, or wake-word recognition cannot tolerate network latency. Models like those used in hearing-aid apps or push-to-talk features must run locally.
Pose estimation and fitness tracking using camera input requires frame-by-frame inference at 30fps. That is 1,800 inferences per minute — impractical and expensive in the cloud, but routine on-device with optimized models.
When Cloud Inference Is the Right Choice
Cloud inference wins when the model is too large to run on a phone, when the task requires frontier reasoning capabilities, or when the feature is used infrequently enough that the cost-per-call model makes sense.
Large language model interactions — chatbots, content generation, complex summarization — require models with billions of parameters. Even aggressively quantized, these models exceed practical on-device constraints. A 7B parameter model quantized to 4-bit still requires roughly 3.5GB of RAM and takes 2-5 seconds to generate a short response on a flagship phone. Frontier models with 100B+ parameters are cloud-only.
Recommendation engines that factor in cross-user patterns inherently need server-side data. Collaborative filtering, trending analysis, and personalization models that learn from aggregate behavior cannot run in isolation on a single device.
Fraud detection benefits from real-time access to global pattern data. While some on-device heuristics can flag suspicious activity, comprehensive fraud models need server-side context that individual devices do not have.
Technical Constraints for On-Device Models
Running models on-device imposes hard constraints that shape your model selection and optimization strategy.
Model size is the primary bottleneck. App store guidelines and user expectations limit what you can bundle. Apple recommends keeping initial app downloads under 200MB. A typical on-device ML model should stay under 50MB — ideally under 20MB — to avoid bloating the app. This rules out large models unless you download them post-install.
Quantization is essential. Converting a model from 32-bit floating point to 8-bit integer (INT8) reduces size by 4x and improves inference speed by 2-4x on mobile hardware with minimal accuracy loss — typically under 1% degradation for classification tasks. Going to 4-bit quantization halves the size again but introduces more noticeable accuracy tradeoffs. Tools like Core ML's compression utilities, TensorFlow Lite's post-training quantization, and ONNX Runtime's quantization toolkit make this accessible.
ONNX Runtime deserves specific mention as a cross-platform inference engine. It runs on iOS, Android, Windows, macOS, and Linux with a single model format. If you are targeting both platforms and want to avoid maintaining separate Core ML and TFLite models, ONNX is a strong choice. Performance is within 10-20% of native frameworks on most tasks.
Battery and thermal impact is a real concern for on-device inference. Running a neural network on the GPU or Neural Engine generates heat and consumes power. Continuous inference (processing every camera frame, for example) can drain 5-15% battery per hour depending on model complexity and hardware. Design your feature to run inference only when needed — trigger on user action rather than running continuously.
The Hybrid Approach
The most effective mobile AI architectures often combine both approaches. The pattern is straightforward: run fast, private, and frequent inferences on-device, and escalate to the cloud for complex reasoning.
A practical example: a customer support app runs intent classification on-device to instantly categorize the user's message (20ms, no network). Simple intents like "check order status" trigger local UI flows. Complex intents that require nuanced response generation get routed to a cloud LLM. This architecture handles 60-70% of interactions without a network call, reducing cloud costs and improving perceived responsiveness.
Another pattern: on-device models handle initial processing — face detection, document edge detection, speech-to-text — and send the extracted, structured data to the cloud for higher-level analysis. This minimizes data transmitted, reduces latency for the initial user feedback, and keeps raw biometric or document data off your servers.
Making the Decision
Start with three questions. Does the feature need sub-100ms response time? Does it process sensitive data you would rather not transmit? Will it run more than a few times per session per user? If you answer yes to any of those, evaluate on-device first.
If the task requires a model larger than 50MB, needs cross-user data, or involves frontier-level reasoning, start with cloud and optimize from there.
At Apptitude, we build mobile apps that use both approaches — often in the same feature. If you are evaluating where AI inference should live in your mobile architecture, reach out. We can help you benchmark the tradeoffs for your specific use case.