AI inference costs, not training, are reshaping model-serving economics

A frontier model's training run is a number journalists love to quote — hundreds of millions of dollars, tens of thousands of GPUs, months of wall-clock time. It's also, for most AI companies, the wrong number to worry about. Training happens once (or a few times a year, for a refresh). Inference happens on every single user request, forever, and it scales with success: the better a product does, the more it costs to run. That inverts a piece of economics that held for most of software history, where the marginal cost of serving one more user was close to zero.

This is why inference, not training, is the line item actually reshaping how AI companies build products in 2026. The companies that are winning on unit economics aren't necessarily the ones with the best model — they're the ones who've driven down the cost of answering a single query furthest, through a stack of techniques that have matured enough to be standard practice rather than research curiosities.

The arithmetic that makes inference expensive

A transformer's forward pass costs roughly 2 × parameter count × tokens processed in floating-point operations. Training costs roughly three times that per token, because it also runs a backward pass and gradient update — but training happens for a fixed, bounded number of tokens. Inference has no such bound; it scales with every query from every user, indefinitely. A model serving a billion queries a day burns more cumulative compute in weeks than its training run did in months, once a product has real usage.

Context length makes this worse. The key-value (KV) cache that a transformer holds during generation grows with sequence length × number of layers × number of attention heads × head dimension × batch size. Push context windows from 8K to 128K or beyond — which products now routinely do — and the memory required per concurrent request grows accordingly, which caps how many requests a given GPU can serve at once. That cap is what actually sets the cost per query, more than raw FLOPs does.

The techniques doing the real cost-cutting

Quantization is the most direct lever: running inference in FP8 or INT8 instead of FP16/BF16 roughly halves memory bandwidth requirements and can double or more throughput on the same hardware, with carefully managed accuracy loss. INT4 pushes further for models where the accuracy hit is tolerable.

Mixture-of-Experts (MoE) architectures change what "model size" even means for cost purposes. DeepSeek-V3 has 671 billion total parameters but activates only about 37 billion per token — you pay compute for the active subset, not the full model. That's the main reason MoE models can offer frontier-scale capability at a fraction of the inference cost of a dense model with equivalent total parameters.

Attention variants attack the KV-cache problem directly. Grouped-query attention (GQA), used in Llama 3 and many current models, shares key/value projections across groups of attention heads instead of giving every head its own, cutting KV-cache size several-fold with minimal quality loss. DeepSeek's multi-head latent attention (MLA) compresses the cache further by projecting keys and values into a smaller latent space before caching them.

Speculative decoding adds a small, fast "draft" model that proposes several tokens ahead, which the full model then verifies in parallel rather than generating one token at a time. When the draft model guesses right — which it does often enough on everyday text — this cuts latency and GPU-time per output token by roughly half to two-thirds, for no loss in output quality (the big model still has final say on every token).

On the serving-infrastructure side, continuous batching and paged KV-cache management — the technique vLLM popularized as PagedAttention — let a server pack far more concurrent requests onto the same GPU memory by managing cache allocation like a virtual-memory system instead of reserving a fixed worst-case block per request. This is a pure infrastructure win: same model, same hardware, meaningfully more throughput.

Routing: spending the expensive model only when needed

A separate strategy skips optimizing a single model's inference and instead optimizes which model answers each query. A routing layer sends simple, high-confidence queries to a cheap, fast model and escalates only genuinely hard queries to an expensive frontier model. Products built this way report average per-query cost well below what running every query through the top-tier model would cost, because most real-world queries don't need frontier-model reasoning — they need a correct, fast answer.

Character.AI's financial trouble — it filed for Chapter 11 bankruptcy on October 13, 2026 — is a useful case study in what happens when this doesn't get solved. A chat product generates enormous inference volume per user (long, frequent, back-and-forth conversations), and if the per-message inference cost isn't driven down aggressively relative to what users pay or what ads generate, the unit economics simply don't close no matter how large the user base gets.

Specialized inference hardware

General-purpose GPUs remain the default, but dedicated inference silicon — Groq's LPU, Cerebras's wafer-scale engine, SambaNova's dataflow architecture — trade training flexibility for lower latency and lower cost per token on inference-only workloads. These matter most for high-volume, latency-sensitive applications (voice assistants, real-time coding tools) where shaving milliseconds and cents per request at massive scale outweighs the flexibility of a general-purpose accelerator.

Takeaways

If you're building a product on top of an LLM API, track cost per 1,000 output tokens as your core unit-economics metric, not cost per query — context length creep is the silent killer of margins. Set a hard context-length budget per feature rather than letting it grow unchecked. Use a cheap/fast model as the default and escalate to an expensive model only on detected uncertainty or explicit user request for deeper reasoning. And if you're evaluating API providers, ask directly about their quantization, MoE, and caching strategy — a provider running FP8 MoE inference with KV-cache-aware batching can be meaningfully cheaper than one that isn't, for the exact same nominal model quality.