On-Device AI &Edge Inference in 2026

The interesting thing about on-device AI in 2026 is that the platform vendors have stopped pretending. Both Apple and Google now ship small local models and a cloud tier, and both route between them. That tiering is the honest architecture, and it is worth understanding what each tier actually contains before you decide which one your feature belongs in.

Apple published its third generation of foundation models on June 8, 2026. Two of the four tiers run on the device. AFM 3 Coreis a 3B dense on-device transformer trained with quantization-aware training, covering roughly 25 locales. Apple reports it was preferred on 45.6% of text prompts against 23.3% for the 2025 on-device baseline, and reports 61% preference on image understanding. Those are Apple's own human evaluations, not independent measurements, and should be read as directional rather than as a benchmark result.

AFM 3 Core Advanced is the more interesting engineering. It carries 20B total parameters but activates only 1 to 4 billion of them per request, using what Apple calls Instruction-Following Pruning, and it holds the full model in NAND flash rather than resident in RAM. That is the design pattern to notice: on a memory-constrained device you do not win by shrinking the model, you win by reading fewer bytes per token and by keeping the rest of the weights somewhere other than RAM.

On Android, Gemini Nano v3runs through AICore. The reported specifications are 1.8B to 3.25B parameters, a context window around 4,096 tokens, 4-bit quantization, sub-second latency, and no per-inference cost — with a requirement of roughly 12GB of RAM plus a flagship chipset. Treat all of that as secondary-sourced. It comes from reporting and community documentation rather than from Google's own published specification, and you should verify it against Google documentation before you commit a device support matrix to it.

Platform state of play for on-device inference as of August 2026, with evidence quality flagged per row.

The row that matters most to a product engineering lead is the fragmentation one. Pixel 10 and Oppo Find X9 plus 2026 flagships are reported to run Nano v3, while Pixel 9 and the Galaxy Z Fold 7 remain on v2. Your install base is therefore split across model generations you do not control, on a schedule you do not set. Any feature that assumes a single on-device model capability is a feature that will behave differently for a large share of your users, and that difference will surface as support tickets rather than as a metric.

The hardware tier is easy to state. Microsoft's Copilot+ PC floor is 40 TOPS of neural-accelerator throughput. The Snapdragon X2 Elite ships an 80 TOPS Hexagon NPU — double the Copilot+ bar, and up from 45 TOPS on the previous X Elite. Those numbers are how devices get sorted into marketing tiers, and they are genuinely useful for exactly that purpose.

They are close to useless for predicting how fast your model will generate text. TOPS measures peak multiply-accumulate throughput, usually at an unstated precision, in a burst the device cannot hold thermally, on an accelerator your inference stack may not even be using. It is a ceiling on arithmetic. Text generation is not limited by arithmetic.

The reason is structural, and it splits cleanly into two phases. Prefill — processing the prompt you already have — is compute-bound and parallel, and this is where an NPU genuinely earns its transistors, along with vision and audio encoders. Decode — generating one token at a time, each conditioned on the last — is serial, and every single token requires reading the active weight set out of memory. There is nothing to parallelize across, so the arithmetic units sit idle waiting for memory.

That gives you a back-of-the-envelope identity worth writing on a whiteboard before any hardware decision: sustained decode throughput is capped by memory bandwidth divided by the bytes of active weights read per token. Not by TOPS. Which is why a sparsely activated design like AFM 3 Core Advanced, reading only 1 to 4 billion parameters per request out of 20 billion total, is a bandwidth optimization first and a capacity trick second.

This is the section product teams get wrong, and it is the reason on-device features so often demo beautifully and ship badly. There are four physical constraints, they compound, and none of them appear on a spec sheet next to the TOPS figure.

Memory bandwidth is the dominant one. Independent measurement puts mobile memory subsystems at roughly 50 to 90 GB/s against 2 to 3 TB/s on a datacenter GPU. That is a 30 to 50 times gap, and because decode reads the active weights once per token, it translates almost directly into a throughput ceiling. No NPU headline number closes it. No compiler optimization closes it. It is the shape of the hardware.

Thermal throttling is the one that ruins demos. Sustained inference costs 20 to 40% of speed after 10 to 15 minutes. Phones have no active cooling; thin laptops have very little. The consequence for engineering practice is specific: a benchmark run on a cold device tells you about minute zero and nothing about minute twelve, and minute twelve is where your user actually is when they are getting value from the feature.

Battery is the one that shows up in reviews. Laptops burn 30 to 60% of battery per hour during active inference. Phones are constrained harder by thermal envelope than by cell capacity, but the user experience is the same complaint. A local model running continuously in the background is not a feature, it is a support category.

And the measured throughput numbers are what they are. A 7B model on an 8GB laptop delivers roughly 10 to 25 tokens per second. An Apple M3 delivers roughly 50 to 80. A streaming interface can hide 15 tokens per second reasonably well. A batch job that summarizes forty documents cannot.

The four physical constraints on on-device inference, with measured figures from independent testing (arXiv 2603.23640) as of 2026.

Benchmark hot, on battery, on the slowest device you intend to support. Every number improves when you test cold, plugged in, on a flagship — which is exactly why that is the test everyone runs and nobody should trust.

— Frenchy Digital engineering principle

One more constraint that is easy to miss until integration: the model competes with your application for memory. Weights plus KV cache are resident, and on mobile the operating system does not kill the model, it kills your process. Long context is expensive twice — once in bandwidth per token, once in resident memory — which is a large part of why on-device context windows are small and will stay small.

The local runtime ecosystem has consolidated around three projects, with the platform frameworks as a fourth option that most shipping mobile apps should consider first. GitHub star counts below are live figures as of August 10, 2026 — they measure attention, not fitness for your build, and should be read that way.

Local inference runtimes, live GitHub data as of August 10, 2026. Sources: Ollama, llama.cpp, MLX.

On the model side, the small-model class has improved considerably and the improvement is real even if the published numbers are not trustworthy in detail. The sub-3B class moved from under 50 MMLU in 2023 to roughly 65 to 70 in 2026. That is a genuine shift in what a phone-resident model can do. It is also measured on a benchmark that has been retired for frontier comparison because it saturated, so treat it as evidence that the class improved and not as a selection criterion.

Small-model options for on-device deployment. All benchmark figures above are secondary-sourced and should be verified against your own task before selection.

The architectural pattern worth extracting from that table is Gemma 3 E4B: 8B total parameters with 4B activated per token. Combined with Apple's 20B-total, 1–4B-activated design, the direction is unambiguous. On memory-bandwidth-bound hardware, the winning move is to hold more parameters than you read. Expect more of the interesting on-device architectures to be sparsely activated rather than simply small.

Four-bit floating point is where quantization currently sits. The two formats in contention are NVFP4, which uses 16-element groups with FP8 E4M3 scales for an effective 4.5 bits per element, and MXFP4 at 4.25 bits per element with roughly 15% higher throughput on B200 hardware. Note the qualifier on that throughput figure: it is measured on a datacenter GPU. What matters for device work is that the tooling, kernels, and calibration recipes are maturing around four-bit floating point generally.

The accuracy story is where the on-device implication lives, and it is not the one the format announcements lead with.

Four-bit floating-point formats and accuracy recovery against BF16 by model size. Source: Red Hat Developers, February 2026.

This is why quantization-aware training matters as much as it does for the platform models. Apple states that AFM 3 Core was trained with QAT rather than quantized after the fact, which is a materially different proposition from taking an open-weights checkpoint and running post-training quantization over it. If you are quantizing a model yourself, the honest expectation is that you land nearer the bottom of that 95–98% band, on a benchmark, and that your own task may behave worse than the benchmark did.

The practical instruction is short: never ship a quantization decision without measuring it on your own evaluation set. A three-point drop on MMLU is an abstraction. A three-point drop in structured-output validity, or in tool-argument correctness, is an outage.

Here is the framework we use on engagements. It is deliberately unsentimental about on-device, because the failure mode we see most often is a team that decided on the architecture before they decided on the requirement.

On-device versus cloud inference by requirement — Frenchy Digital decision framework, 2026.

Two entries deserve expansion because they are the ones most often argued badly.

Zero marginal cost is real, and it arrives later than people expect. The device path has a large fixed cost — model selection, quantization and its evaluation, packaging, an update mechanism, a per-device QA matrix, thermal and battery testing, and a permanent support category — and a marginal cost of approximately zero. The cloud path is the reverse. That means the comparison is a payback period, not a unit-cost comparison, and the payback period is measured in months of engineering salary against months of API invoices.

Latency is two different numbers. On-device wins decisively on time to first token, because there is no network. It frequently loses on time to last token, because 10 to 25 tokens per second on a laptop is slower than a hosted small model streaming over a good connection. For inline completion, live captioning, or a camera overlay, first-token latency is the whole experience and local wins. For a long summarization, total completion time is the experience and cloud usually wins.

The strongest argument against pure on-device architecture is that the company with the most commercial incentive to build one did not. Apple's third-generation stack has four tiers: AFM 3 Core and AFM 3 Core Advanced on the device, and a server tier of AFM 3 Cloud — a Parallel-Track mixture-of-experts running on Apple silicon — and AFM 3 Cloud Pro on NVIDIA GPUs in Google Cloud. Apple reports Cloud Pro at +10% on text and +14% on image over Cloud. The company that markets privacy hardest still escalates, and still rents NVIDIA capacity to do it.

Take that as the reference architecture rather than as a compromise. A hybrid design gets you the privacy and latency properties of local inference on the common case, and frontier capability on the cases that need it. What it costs you is a routing layer that has to be designed as deliberately as the models on either side of it.

Hybrid on-device and cloud routing architecture — component breakdown, Frenchy Digital 2026.

One design detail that saves considerable pain later: keep the routing decision separate from the prompt. A router implemented as an instruction inside a system prompt is a router that untrusted content can influence. A router implemented as code, reading typed signals — task type, content length, thermal state, network availability, user preference — is a router you can test, log, and reason about.

Concretely, this is the layering we deploy. It is deliberately unexciting; the interesting parts of an on-device system are the parts that fail on somebody else's phone.

• 1. Capability probe at launch: Detect the device class once per session: available RAM, chipset, OS version, platform model availability and generation. Cache it. Every downstream decision reads this rather than re-detecting, and it is the single source of truth for the degradation path.
• 2. Typed request envelope: Every AI request carries a declared task type, an input size, a sensitivity flag, and a user preference. The router reads fields, never free text. This is what keeps untrusted content from influencing where the content goes.
• 3. Policy router in code: A deterministic function from envelope plus device capability plus network state plus thermal state to a destination. Testable, loggable, and reviewable in a pull request. No model decides where data goes.
• 4. Local inference path: Platform model where possible, embedded runtime where not. Bounded context, bounded output length, a hard timeout, and a cancellation path the user can reach.
• 5. Escalation gate: The only place a request can cross to the cloud. It applies the sensitivity flag, the user preference, and any regulatory rule, and it emits an event whether or not it escalates. One gate, one audit point.
• 6. Cloud inference path: A hosted model with pinned version, batch and caching applied where the workload allows, and its own cost ceiling. Treat a model upgrade like a schema migration: canary it and watch quality metrics.
• 7. Output validation: Structured-output schema validation, refusal detection, and length checks run identically on both paths, so a downstream consumer cannot tell which tier produced the result. This is also the confidence signal the router learns from.
• 8. Offline queue: Work that requires cloud and has no network either degrades explicitly or queues with visible state. Silent failure is the option that generates support tickets.
• 9. Aggregate telemetry: Latency, throughput, escalation rate, validation failure rate, thermal state, and battery delta — as aggregates, opt-in, with no content. You need the evaluation signal without recreating the disclosure the architecture removed.
• 10. Kill switch and remote configuration: The ability to disable a path, shift the routing threshold, or force cloud-only without an app release. This is the control that turns a bad model generation from an emergency submission into a config change.

The cost argument for on-device is straightforward to state and easy to get wrong. Local inference has a marginal cost of roughly zero to you — the user pays in battery and storage — against a cloud path billed per token. The crossover is the point where accumulated API spend exceeds the fixed engineering cost of the device path. To find it you need both sides, and most analyses only build the side that arrives as an invoice.

Start with the cloud side. These are list prices per million tokens, verified against official pricing pages and current as of August 2026. Prices in this market change monthly, so re-verify before you commit a number to a business case.

Selected API pricing as of August 2026. Sources: Anthropic, OpenAI, Google, Together AI.

For a concrete shape, Anthropic publishes a worked example: a customer-support conversation at roughly 3,700 tokens on Haiku 4.5 costs approximately $37 per 10,000 tickets. That is a vendor-published example rather than an independent measurement, but the arithmetic is checkable against the price table and it gives you a realistic order of magnitude. At that shape, a million conversations a month is a few thousand dollars — a real number, and a smaller number than one senior engineer-month.

Three adjustments that materially move the cloud side before you compare:

• Batch and caching discounts: Anthropic's Batch API is 50% off both input and output and stacks with prompt caching, where a cache read costs 0.1x base input. If your workload tolerates latency or repeats a long prefix, the effective price is well below list.
• The tokenizer shift: Claude 4.7 and later, including Sonnet 5, use a tokenizer that produces roughly 30% more tokens for the same text. The per-token price did not change; the cost per request did. Re-baseline with a token-counting endpoint rather than applying a multiplier to an old estimate.
• Reasoning tokens bill as output: On both major providers, reasoning tokens are billed at the output rate — the expensive side. A model configured with a high reasoning effort can cost several times a naive estimate built from visible response length.

Now the device side, which does not arrive as a bill and is therefore systematically under-counted: model selection and quantization plus the evaluation to prove it did not break anything; packaging and an update mechanism; a per-device QA matrix including thermal and battery testing on physical hardware; a degradation path for devices below the floor; and a permanent support category for battery, storage, and device-specific failures. In our scoping, that is the $30k–$80k band for a single feature, and it recurs at a lower rate every year the feature exists.

Which is why the durable arguments for on-device are the non-cost ones. Privacy, offline operation, and latency floors do not erode when someone cuts a price. Pure cost arbitrage does.

Benchmark scores are unreliable across the board in 2026, and they are least reliable for exactly the model class this article is about. Small models are the ones most likely to have been tuned toward public benchmarks, and public benchmarks are the ones most thoroughly contaminated. If you select an on-device model from a leaderboard, you are selecting on the noisiest available signal.

Why published benchmark scores mislead for on-device model selection, with primary findings. Sources: contamination analysis, UC Berkeley RDI, SWE-rebench.

The contamination finding is worth stating in full because it is the cleanest evidence available. On decontaminated task sets collected after model training cutoffs, frontier models land dramatically below their scores on the older public benchmarks — roughly three times better on the contaminated set, and about six times better at locating the relevant edited files without project context, which is the kind of result that is difficult to explain by capability. If contamination distorts frontier results that much, it distorts small-model results at least as much.

The reward-hacking finding is more alarming and less discussed. A UC Berkeley team reached approximately 100% on Terminal-Bench, SWE-bench Verified, SWE-bench Pro and WebArena, roughly 98% on GAIA and 73% on OSWorld, without solving a single task — via test-harness hooks, configuration leakage, prompt-injected judges and environment-state manipulation. A follow-up catalogued 219 distinct flaws across ten benchmarks. A benchmark number, absent a description of the harness that produced it, carries almost no information.

Report reliability as pass^k rather than pass@1. On the tau-bench retail environment, GPT-4o measured under 50% pass^1 and roughly 25% pass^8 — a reliability collapse across repeated trials that a single-attempt score hides entirely. For a device feature that a user invokes dozens of times a day, pass^k is the number that describes their experience.

Everything above becomes concrete the moment you have to ship it inside an app that goes through store review. Here is the checklist we work through with mobile and product engineering leads, in the order the decisions actually have to be made.

On-device AI decisions specific to shipping mobile applications — Frenchy Digital build checklist, 2026.

The single highest-leverage decision on that list is the first one: whether you ship weights at all. Using the platform-provided model — Apple's on-device foundation models, or Gemini Nano through AICore — means no weights in your bundle, no separate update path, no download funnel, and the platform vendor owning the thermal and battery behavior that would otherwise be your review problem. The cost is that you get the model the platform gives you, on the generations it supports, with the fragmentation described earlier.

Shipping your own weights buys control and costs everything else. It is the right call when you need a specific fine-tuned behavior, when you need identical behavior across iOS and Android, or when the platform model does not cover your locales. It is the wrong call when the driver is that a benchmark said your chosen open-weights model scores higher, because see the previous section.

The second-highest-leverage decision is the degradation path. Define, before design, what the feature does on a device below the RAM or chipset floor. Three options are defensible — escalate to cloud, offer a reduced version, or show an explicit unsupported state — and one is not, which is letting the feature silently fail. Half your install base may sit below the floor, and on Android that is not a hypothetical.

Add power draw and sustained thermal behavior to the release checklist next to crash-free rate. An AI feature that ships with an unmeasured battery cost is an AI feature that ships with an unmeasured retention cost.

— Frenchy Digital mobile principle

Every one of these has appeared in a real vendor evaluation, an internal architecture review, or a published article we had to correct. None are hypothetical.

The Frenchy Digital red-flag list for on-device AI claims, 2026.

Ask for one artifact instead of a deck: a sustained throughput and battery measurement, on your floor device, on battery, after fifteen minutes of continuous use. A vendor who cannot produce it has not run it, and a vendor who has not run it does not know what they are selling you.

— Frenchy Digital buyer principle

These are the bands Frenchy Digital uses to scope on-device and hybrid inference work in 2026. They assume measurement on real hardware is in scope from the start, because retrofitting a device matrix and a thermal test plan after the feature is built is what makes these projects overrun.

Frenchy Digital cost bands for on-device and hybrid AI engagements, 2026.

Senior-led delivery runs $150 to $225 per hour, and ongoing retainers run $2,500 to $9,500 per month covering model and runtime upgrades, evaluation-set expansion, device-matrix maintenance as new hardware ships, incident response, and a quarterly technical review. Every engagement carries a 30-day post-launch warranty, and you receive a written scope with a fixed-price phased proposal within 5 business days of the discovery call.

A budgeting note that surprises people. The discovery band is the one to spend on first and the one teams most often skip. Two to four weeks of measurement on real devices routinely establishes that the intended feature is either comfortably feasible or clearly not, and it does so before anyone has committed six figures to an architecture. We have run discovery engagements whose entire output was a recommendation not to build the on-device path — and that was the highest-return work in the relationship.

On-device inference is a genuine capability with a narrower application than the marketing implies. If you are building a business case, build it on the following rather than on a keynote.

• The bandwidth gap is not closing on your timeline: A 30 to 50 times difference between mobile and datacenter memory bandwidth is a property of the hardware class, not a software problem awaiting a fix. Sparsely activated architectures reduce bytes read per token, which helps at the margin. They do not change the order of magnitude.
• The cost case erodes over time: Cloud inference prices for a fixed capability level have fallen at rates measured between 9x and 900x per year (Epoch AI, published March 2025; no 2026 update verified). Any crossover analysis with a multi-year payback is competing against a price curve that moves faster than device hardware does.
• Quantization hurts the size class you are forced into: Accuracy recovery against BF16 is roughly 95 to 98 percent at 7B to 14B, against roughly 99 percent at 70B and above. The device is the hardest place to quantize and the place you have no choice.
• Fragmentation is permanent: Gemini Nano v3 is reported to require roughly 12GB of RAM plus a flagship chipset, with older flagships remaining on v2. Your install base will always span multiple on-device model generations, and the split is set by device replacement cycles rather than by your release schedule.
• Locality is not a security control, and prompt injection is not solved: As of August 2026 there is no mitigation that makes an agent safe against prompt injection; the correct framing is defense in depth and blast-radius reduction. Running locally changes the delivery vector — hostile instructions arrive in a shared document, a web page, or a calendar invite instead of through your API — and a local model with access to contacts, files and messages has a larger blast radius than a sandboxed cloud agent. Allowlist tools, reject arguments the user never supplied, and treat retrieved content as data. Map against the OWASP Top 10 for LLM Applications.
• Vendor preference numbers are internal evaluations: Apple's 45.6 percent versus 23.3 percent text preference and 61 percent image-understanding preference are Apple's own human evaluations against its own prior baseline. They are informative about generational improvement and say nothing comparative about other vendors' models.
• The Gemini Nano specifications are secondary-sourced: Parameter counts, context window, quantization and RAM requirements for Nano v3 come from reporting rather than from Google's published specification. Verify against Google documentation before any of it enters a device support matrix or a customer commitment.
• Benchmarks measure the happy path: Published scores are produced on curated tasks, in controlled harnesses, on cold hardware. None of the failure modes that determine whether your feature ships — thermal degradation, memory pressure, a partial download, a device below the floor — appear on any leaderboard.
• There is no public dataset for your battery question: The 30 to 60 percent per hour figure is an independent measurement across a sample, not a prediction for your workload on your device matrix. Nobody can give you your number except your own instrumentation.

None of this argues against on-device inference. It argues for deciding on the basis of a requirement rather than an aesthetic, measuring on real hardware before committing, and building the hybrid path so that a wrong guess about the model tier is a routing change rather than a rewrite.