Local AI has caught up — and the cloud doesn't know it yet.

The 18-month gap between OpenAI-class models and what runs on your laptop has collapsed. We've been benchmarking the new generation of on-device models for six months. Here's what we found, why it matters for your stack, and the part of the architecture that still actually decides quality.

Why we're writing this

For three years, the default assumption in product engineering has been: real AI lives in the cloud. You hit an API, you pay per token, you pray latency stays under a second, you eat the privacy review when legal pushes back. Local models were toys — Stable Diffusion if you wanted images, Llama if you were a hobbyist with a 4090.

That assumption is wrong as of about eighteen months ago. It's more wrong every quarter. The teams still defaulting to cloud APIs for anything that fits in 8 GB of VRAM are spending money and giving up control they don't have to.

This piece is the technical case, with numbers we measured ourselves and what we changed in NYMPH because of it.

The cloud assumption, briefly

The cloud-by-default thesis goes like this:

1. Capability: only frontier labs can train models that don't embarrass you.
2. Hardware: consumer GPUs can't hold a useful model.
3. Quality: small models hallucinate, miss instructions, can't follow long context.
4. Updates: you want the latest model, you want it now, you don't want to redownload.

Each of those was true at some point in 2023. Each of them is now either false or no longer the bottleneck.

What changed: three quiet shifts

1. The architectures finally generalized.

The post-Llama-3 generation — Llama 3.1 / 3.2, Qwen 2.5, Mistral Nemo, Phi-3.5, Gemma 2 / 3, the 3-to-9-billion parameter band — were trained on an order of magnitude more tokens than the same parameter count was a year prior. Quality scales with data, not just parameters. A modern 8B model trained on 15 trillion tokens beats a 2023-era 70B model trained on 2 trillion on most public benchmarks. We confirmed that on internal eval suites too.

2. Quantization became actually lossless (for product use).

4-bit quantization in 2023 was a 5–8 point hit on MMLU. With activation-aware methods (AWQ, GPTQ-3.0, K-quants in llama.cpp) and the new GPU-native int4 tensor cores in 2024+ Apple Silicon and NVIDIA, we measure typically 1–3 points across our benchmark battery — well inside the noise of API model swaps you're already living with.

3. The harness now matters more than the weights.

This is the one most teams miss. A small model with a good runtime — tool execution, retrieval, response verification, conversation compaction, structured output enforcement — outperforms a frontier model with naïve prompting on real tasks. We've published this result before; it's also visible in everyone else's stacks (Cursor's evals, Devin's loop, Claude Code's tool use).

The functional comparison that actually matters

Benchmarks are easy to game. We care about what hits production. Below is what we see across the use cases we ship:

Read that table the right way: for the workloads most products actually run — chat, structured output, code, document Q&A, agentic tool calls — local already wins on cost and tie-or-better on quality. Cloud retains a real lead at the absolute frontier. The question is whether your product is using the frontier, or just defaulting to it because you started there in 2023.

What "adaptable" actually means now

The argument for cloud always included "and we can swap the model when something better comes out". That's still true. It's also true now of local:

• Ollama, LM Studio, llama.cpp all hot-swap models in seconds — drag a new GGUF, point your runtime at it.
• Same model file format across consumer and server hardware. The binary you ship to a laptop runs on a workstation, runs on a homelab box, runs on a cloud GPU if you want to host it yourself.
• Tool definitions are model-agnostic. JSON schemas for function calls work identically against local and cloud — your harness layer doesn't care.
• Fine-tuning is two-day cycle, not two-month. LoRA adapters on a single 4090, merged at runtime. No platform's permission needed.

"Adaptable" used to mean "the cloud vendor is shipping new models". It now means you can move faster than the cloud vendor — because you're not waiting on a release calendar or a price card.

The honest version: cloud APIs aren't dying. They're getting demoted from default to fallback.

Where this lands for your architecture

The shift we recommend is unsexy: it's a routing decision, not a rewrite. For each AI call your product makes, ask:

1. Does this need to run with no internet? (offline-capable products → local)
2. Does this carry data we can't legally send to a vendor? (regulated industries → local)
3. Is this a hot path with strict latency? (real-time UI → local)
4. Is this dominated by token cost? (high-volume background tasks → local)
5. Is this genuinely at the frontier of capability? (research-grade reasoning → cloud, for now)

In practice we route 80–90% of NYMPH's traffic to the local NYMPH AI 4.5B model and reserve cloud APIs (Claude, GPT-4) as second-opinion calls for the user when they explicitly ask for them. Cost-per-active-user collapses by an order of magnitude. Latency drops to single-digit milliseconds for the first token. Privacy review gets boring instead of interesting.

The catch nobody mentions

Local AI isn't free, it's front-loaded. You eat the cost in three places:

• Distribution. The model is 4–8 GB. You're shipping a binary. You need an installer, a way to update, a way to bundle. (We solved this with a one-line installer.)
• The runtime. A model file is not a product. The harness — tool routing, output validation, memory, retrieval, error recovery — is the actual engineering work. That's why we say "value lives in the runtime, not the weights".
• Hardware floor. You need ~8 GB of unified memory or VRAM. That excludes the cheapest devices, but covers every Apple Silicon Mac since 2020 and every modern Windows laptop with discrete graphics.

None of those are fundamental. All of them are tractable engineering. None compare to "we're sending a regulated user's medical history to a third-party API to summarize it."

What we built around this thesis

NYMPH Pulse is the product version of this argument. The chat, the agent, the workspace, the activity log — all of it runs against a 4.5B-parameter NYMPH AI model that lives on the user's machine. The cloud is there as a courtesy: when a user wants to compare against Claude or GPT-4 for a specific question, they can. By default, no token leaves their device.

Pricing reflects the architecture. There's no per-token meter because there are no per-token costs to pass on. $99/year covers a year of model updates. $149.99 once covers everything for life.