If your AI feature costs more every month and nobody can say why, the cause is almost always the same: your app sends the model far more text than the answer needs, and sends it again on every turn. The fix is a short sequence. Measure where tokens go, turn on caching, trim what you send, cap what comes back, and only then route easy work to cheaper models. I'm Hanad, a senior engineer in Vienna, and I build and rescue fixed-price software that includes this kind of LLM cost work. Below is the sequence in the order I would apply it, with the numbers from the providers' own documentation.
Key Takeaways
- A token is a chunk of text the model reads or writes. You pay for input, output and sometimes hidden reasoning tokens, and the bill follows the provider's own count.
- In chat and agent features, the same instructions and history are resent every turn, so cost grows much faster than the number of messages.
- Prompt caching comes first. Cached input is billed at a fraction of the normal rate, and Anthropic's guidance puts it before every other lever.
- Measure a baseline before changing anything, and keep a small set of test questions so you can prove quality did not drop.
- Routing to cheaper models comes last, because it depends on having a working quality check.
Table of Contents
- What a token is and where yours go
- Why the bill grows faster than usage
- Step 1: Measure before you change anything
- Step 2: Turn on prompt caching
- Step 3: Send less input
- Step 4: Cap what comes back
- Step 5: Route easy work to cheaper models
- Check that quality held
- Why prototypes built in Lovable or Bolt leak tokens
- If you want this done for you
- FAQ
- Sources
What a token is and where yours go
A token is a small piece of text, often a word fragment. OpenAI's documentation gives a rough rule for English: one token is about four characters, or 0.75 words. The same page notes that total tokens drive cost, latency and whether your request fits the model's context window, which is the maximum amount of text the model can read at once.
That rule is only a rough guide. An academic study of tokenizers found that compression ratios differ by 20 to 50 percent across domains and models, and by up to 100 percent for emoji and numbers. The same text can cost a different number of tokens on different providers, so the only count that matters is the one on your invoice.
Every request has several kinds of tokens:
- Input tokens: everything you send. That means your instructions, the conversation so far, retrieved documents, tool definitions, tool results and the user's latest message.
- Output tokens: what the model writes back, including the arguments it passes to tools.
- Cached input tokens: input the provider has seen before and bills at a lower rate.
- Reasoning tokens: internal thinking tokens. OpenAI's help center says they count toward output usage and are billed as output.
One more detail trips people up. Chat models add formatting tokens for message roles and boundaries, so counting your text alone undercounts the real input. OpenAI recommends its input-token counting API when you need the full figure for a request.
Why the bill grows faster than usage
In an agent or a chat, each turn resends everything before it. Anthropic's cost guidance spells out the consequence: a 40-turn task sends its first turn 40 times, so task cost grows roughly with the square of the turn count. Doubling the length of a conversation more than doubles what you pay for it.
This is why a prototype that felt cheap in a demo gets expensive with real users. Real users have longer sessions, upload more, and trigger more tool calls. Each of those adds to the pile that gets resent.
Step 1: Measure before you change anything
Do not trim blindly. You need a baseline, and Anthropic's cookbook recommends the same order: get the task working on a capable model, build an evaluation, measure a baseline, then apply savings in sequence. You can read that sequence in the Claude cost optimization cookbook.
What to record on every model call:
- Input tokens, output tokens and cached tokens, as reported by the provider in the response.
- Which feature, user action and model produced the call, so you can see which one is expensive.
- A breakdown of the input by layer: instructions, history, retrieved documents, tool definitions. Firecrawl's token guide recommends tagging usage by feature, task and model, and counting before you send.
Then look at where the weight sits. In my experience of reading prototype code, the surprise is rarely the user's message. It is a long instruction block, a full document pasted in where one paragraph would do, or a list of tool definitions that never changes but is sent every time. If you want a deeper walk through reading your context layer by layer, I wrote a guide to auditing LLM context management.
If you also want a price baseline to compare against, it helps to understand how reserved AI inference capacity pricing is structured, since your per-token rate is only half of the cost equation.
Step 2: Turn on prompt caching
Caching is the biggest single lever for most products, and it asks very little of your code. The idea is simple. When many requests start with the same text, the provider reuses the work it already did for that prefix and charges less for it.
The provider details:
- OpenAI: caching works when requests share the same prompt prefix. It is enabled by default for supported models, with cached input discounted by up to 95 percent, and the docs describe a dashboard for hit rates.
- Anthropic: its guidance says to turn caching on before any other lever. Each cached resend costs about a tenth as much on older models, with a smaller fraction on newer ones, and each turn pays a 1.25x write rate only for new content (see the Anthropic cost guide).
- Minimums: according to Firecrawl's guide, OpenAI caching applies automatically to prompts of 1,024 tokens or more, and Anthropic has a 512-token minimum on the models it covers.
A worked example
Here is my own arithmetic, not a measurement. Say your instructions are 3,000 tokens and a session runs 20 turns. Without caching you pay for 3,000 tokens 20 times: 60,000 input tokens for the instructions alone. With caching at the Anthropic pattern of one 1.25x write and nineteen reads at about one tenth, the same instructions cost about 3,750 plus 5,700, or 9,450 token-equivalents. That is roughly 84 percent less for that part of the request.

Keep the prefix stable
Caching only works if the start of your prompt does not change. The rules, per Firecrawl's cache-stability advice:
- Put static content first and dynamic content last.
- Never put a timestamp in the system prompt.
- Do not add, remove or reorder tools mid-session, because tool definitions sit in the prefix.
- Do not switch models mid-session, because caches are per model.
A single changing line near the top of a prompt can quietly turn every request into a cache miss. This is the most common reason I find for a cache that exists on paper and saves nothing. For a longer engineering treatment, see my write-up on prompt caching.
Step 3: Send less input
Anthropic's guidance says most agent requests carry tokens that never influence the answer, and trimming them rarely costs output quality. Its suggestions include filtering fetched web pages, resizing images, deferring tool definitions that are not needed yet, and calling tools programmatically so only filtered results enter the context. On agentic search benchmarks, the programmatic approach used 24 percent fewer input tokens with a higher score.
OpenAI's latency guide gives matching advice: filter search results for relevance, strip markup you do not need, and send only the context required to answer accurately.
Practical ways to do this:
- Cut retrieval down. If your app pastes in five documents, test whether two or three relevant passages give the same answer.
- Strip the packaging. HTML, repeated headers, signatures and navigation text are tokens you pay for and the model ignores.
- Trim history. Keep the last few turns in full and replace older ones with a short summary.
- Remove duplicate instructions. Prototype prompts tend to collect the same rule three times, added after three different bugs. Repeated and irrelevant context is a common problem in agent workflows, and a list of common AI prompts for business shows how much boilerplate prompts tend to carry.
The pattern to avoid is compressing everything at once. Change one thing, run your test set, and keep the change only if answers hold.
Step 4: Cap what comes back
Output tokens are usually priced higher than input tokens, and the model's answer also takes time to generate. OpenAI's guidance covers the main controls:
- Ask for the level of detail you need, for example one sentence for the next step.
- Lower verbosity or reasoning effort on models that support it.
- Request only the fields your app uses in structured output.
- Set a maximum output budget. A low cap can leave a response incomplete, so check the response status before using a partial answer.
These are all described in the latency optimization guide. The last point matters in practice: a hard cap with no status check produces answers that stop mid-sentence, which users notice immediately.
Step 5: Route easy work to cheaper models
Once caching, input and output are under control, you can send simple requests to a smaller model. Anthropic's sequence puts this last: select the cheapest model tier that clears your quality bar, which means you need the bar first.
Two cautions from the research:
- Per-token price is a poor comparison. One measurement study reported by an industry analysis found the same 2,888-character TypeScript file produced 681 tokens on one tokenizer and 1,178 on another, so the sticker price tells you little until you measure your own content.
- Switching has a cache cost. Because caches are per model, switching mid-session can cost more than staying put. Route at the start of a task, not in the middle of a long conversation.
An academic measurement study of seven token-saving tactics on coding-agent workloads found that local routing plus prompt compression saved 45 to 79 percent of cloud tokens on edit-heavy and explanation-heavy work, and that the best mix of tactics depends on the workload. Your workload is not theirs, which is why Step 1 comes first.
Check that quality held
Savings that break your product are not savings. Keep a fixed set of real questions, with the answers you consider good, and run it after every change. Compare cost and quality side by side.
OpenAI's guidance also suggests measuring time to first token and end-to-end latency at the median and the 95th percentile on representative traffic, and recording input, cached and output tokens alongside the timings. Tokens and speed usually improve together, so one log serves both goals.
A note on automated judging: the Local-Splitter study used a small model as a judge and found its verdicts were inconsistent on 17 of 40 pairs. If you use a model to grade answers, spot-check its grades yourself.
For ongoing visibility after launch, my guide on shipping LLM monitoring covers what to watch, and the two-week LLM cost playbook lays out how I approach the cost side as a project.
Why prototypes built in Lovable or Bolt leak tokens
Prototype builders optimize for getting something running. That is the right goal at that stage. The side effect is that prompts get long, history is never trimmed, tool lists are sent whole, and nobody logs usage. None of that matters at ten test users. It matters at a thousand.
If this is your situation, the token bill is usually one symptom among several: the app is slow, logins or payments are fragile, and no developer wants to touch the code. Fixing prompts in a codebase that cannot be maintained is a poor use of money. It is better to treat the prototype as the specification and rebuild the parts that carry load.
What I would do in your position
I build one thing at a time, end to end, and I do it myself. When a founder brings me an AI feature with a growing bill, I start with the usage numbers, because they show what is expensive before anyone guesses.
The fixed-price format matters here. Cost work has no natural end if it is billed by the hour, and a founder pays for exploration. A fixed scope forces the useful question: which two or three changes move the bill, and what proves they did not hurt quality?
You own the code from the first commit, and I use a deliberately plain stack, TypeScript, Next.js, Node and SQL, so the next developer can read it. Token logging, caching and trimming are only worth having if someone else can maintain them later.
Hanad Kubat
If you want this done for you
I build working software for founders and owner-operated businesses at a fixed price, from €12,000, in two to four weeks, including AI features with LLM cost optimization built in. You see working progress before you pay, and the code is yours from the first commit. If a prototype is at the point where cost, speed or stability is blocking you, start at hanadkubat.com.
FAQ
What is the fastest way to reduce prompt tokens?
Check whether your static instructions come first in the prompt, then turn on or fix prompt caching. After that, cut retrieved documents and old history down to what the answer needs. These two moves usually remove the most waste without touching answer quality.
How do I know which part of my prompt costs the most?
Log the provider's usage figures for every call, tagged by feature and model. Then break the input into instructions, history, documents and tool definitions. The provider's count is the one that is billed, so use its counting tools rather than a local estimate.
Will fewer tokens make answers worse?
Not if you remove tokens that never influenced the answer, which Anthropic says is true of much of what agent requests carry. Cutting relevant context can hurt, so run a fixed set of test questions after each change and keep only the changes that hold.
Is a cheaper model always the better saving?
No. Token counts differ between providers, caches are tied to one model, and the cheapest option depends on what you send. Apply caching and trimming first, then route to a cheaper model only for tasks your test set shows it handles well.
Sources
- Understanding and counting tokens, OpenAI Help Center
- Advanced usage, OpenAI API documentation
- Prompt caching, OpenAI API documentation
- Latency optimization, OpenAI API documentation
- Optimizing for cost and intelligence, Anthropic documentation
- Cost Optimization on the Claude API, Claude Cookbook
- LLM Token Optimization: 9 Techniques in the Order to Apply Them, Firecrawl
- Local-Splitter: A Measurement Study of Seven Tactics, arXiv
- How long is a piece of string? A brief empirical analysis of tokenizers, arXiv
