The first surprising AI bill usually arrives in month two. The product works, usage grows, and the cost per user turns out to be higher than the price per user.
Most of that is fixable, and most of the fixes are mechanical rather than clever. Here is what actually moves the number.
First: Find Out Where It Goes
Before optimising anything, log input and output tokens per request, tagged by which feature made the call.
Almost everyone is wrong about which endpoint dominates their bill. The expensive one is rarely the flagship feature — it is usually a background job, a retry loop, or a summarisation step that runs on every request and nobody thinks about.
An afternoon of instrumentation reliably beats a week of guessing. Everything below assumes you have done this.
1. Route by Task (Usually the Biggest Win)
Frontier models cost roughly 5–10x what the cheap tier costs. Most applications route everything through one model because it was simpler to build that way.
Split your calls into three buckets:
Simple and high-volume — classification, routing, extraction, tagging, simple rewriting. The cheapest tier handles these well.
Standard production work — most user-facing generation. The mid tier is usually indistinguishable from frontier here.
Genuinely hard — multi-step reasoning, long-horizon agentic work, complex code generation. This is what you are paying frontier prices for.
If 80% of your volume is bucket one and you are running it on a frontier model, moving it down a tier cuts most of your bill without touching quality.
Verify rather than assume. Take fifty real examples, run them on both tiers, and compare outputs. Often the cheaper model is fine and you have been paying for reassurance.
2. Fix Your Prompt Caching
If you send a large stable prefix with every request — a long system prompt, examples, a document — caching makes subsequent requests dramatically cheaper for that prefix.
The mechanic that decides whether it works: caching is a prefix match, and any byte change invalidates everything after it.
Which produces the single most common cost bug in AI applications:
// This makes your entire prompt uncacheable
system: `Current date: ${new Date()}\n\n${LONG_SYSTEM_PROMPT}`The timestamp sits at position zero, changes every request, and invalidates the whole prefix. The fix is ordering — stable content first, volatile content last:
// Cacheable
system: LONG_SYSTEM_PROMPT
messages: [{ role: "user", content: `Current date: ${date}\n\n${question}` }]Other silent invalidators worth grepping for: request IDs or UUIDs early in the prompt, non-deterministic JSON serialisation (unsorted keys), per-user content in the system prompt, and conditionally-included sections.
Verify it is working by checking the cache-read token count in the API response. If it is zero across repeated similar requests, something in your prefix is changing.
3. Stop Sending Context You Do Not Need
Large context windows encourage a lazy pattern: send everything and let the model sort it out. This is expensive and often produces worse results, because relevant context beats abundant context.
Trim conversation history. Most chat applications send the entire history on every turn, which means cost grows quadratically with conversation length. Keep the last N turns plus a summary of what came before.
Retrieve rather than dump. Sending a whole document when three paragraphs are relevant costs the whole document, every time.
Check your prompt for dead weight. Prompts accumulate: instructions for edge cases that no longer exist, examples added during debugging, defensive phrasing nobody has re-read. Prompts written six months ago are usually 30% longer than they need to be.
4. Cap Your Output Tokens Deliberately
Output tokens cost roughly 5x input tokens across providers. This makes response length the most expensive variable you control.
Two levers:
Set a real limit. If your responses should be a paragraph, do not leave the limit at the maximum. This is a ceiling, not a target, but it prevents runaway generations.
Ask for brevity in the prompt. Models default to thorough. A single explicit instruction — "answer in two sentences unless more detail is requested" — reduces output length substantially and often improves the product.
5. Tune Reasoning Effort Per Route
Current models let you control how much reasoning happens before answering. Higher effort spends more tokens on everything, including tasks that did not need it.
The mistake is setting this globally. Your hard reasoning path and your simple classification path should not run at the same effort.
Also worth testing: on simple tasks, high effort sometimes produces worse results, because the model overthinks a problem with an obvious answer. Do not assume more is better — measure.
6. Cache Your Own Results
Distinct from prompt caching: if users ask similar things, cache the answers.
Exact-match caching on a normalised query is trivial to implement and eliminates duplicate work entirely. For repeated questions in a support or docs context, this can remove a meaningful share of calls.
Semantic caching — treating similar-but-not-identical queries as the same — is more powerful and riskier, since a near-miss returns a subtly wrong answer. Use a high similarity threshold and only where the failure is cheap.
7. Batch What Is Not Urgent
Most providers offer batch processing at roughly half price for work that does not need an immediate answer.
Anything on a schedule — nightly summarisation, bulk classification, dataset enrichment, backfills — should probably be batched. It is a straight 50% saving for changing where the request goes.
Common Mistakes
Retry loops with no ceiling. A failure that retries forever burns money silently. Cap attempts, back off exponentially, and log when you hit the limit.
Streaming to nobody. If a request is abandoned — user closes the tab — cancel it. Generation you paid for and nobody saw is pure waste.
Development against production models. Building and debugging against the most expensive tier adds up across a team and a month.
No per-user limit. Without one, a single heavy user or a scripted abuser can produce a bill unrelated to your revenue. This is the failure mode that produces the alarming screenshots.
Summarising things nobody reads. Worth auditing: automatic summarisation steps that generate output no user or downstream process actually consumes.
Cost Per User Is the Number That Matters
Total spend is not the useful metric — cost per active user, compared against revenue per user, is.
If cost per user exceeds revenue per user, growth makes things worse rather than better, and no amount of optimisation fixes a fundamentally inverted unit economic. That is a pricing decision, not an engineering one.
Track it monthly. It is the number that tells you whether your product works as a business.
The Order to Do This In
Instrument. Log tokens per route. An afternoon.
Fix caching. Reorder prompts so the prefix is stable. Often the largest single win.
Route by task. Move simple work to the cheap tier and verify on real examples.
Trim context. History windows, retrieval instead of dumping, prompt cleanup.
Cap output and tune effort per route.
Batch scheduled work.
Add per-user limits. Before you need them.
Steps one to three are where most of the money is. The rest is refinement.
If you are building an AI product independently, our report on 132 indie AI products covers what everyone is actually using, and you can list your product free.
