Short answer: Most AI features are billed per token, and you pay separately for the text you send (input) and the text the model writes (output). The monthly bill depends less on the price list than on how much context you resend with every request. Caching the repeated part of each prompt, capping output, choosing a smaller model where it's good enough and batching work that can wait are the main ways to bring that bill down.

Key points

  • A token is a piece of text. OpenAI's rule of thumb for English is about 4 characters per token, or roughly 75 words per 100 tokens (OpenAI Help Center). Other languages, code and other tokenizers produce different counts.
  • Input, cached input and output tokens are priced differently. On most published price lists, output costs several times more than input.
  • Reasoning ("thinking") tokens are billed as output even though you never see them as answer text.
  • The API doesn't remember your conversation. Each turn resends the history, so input cost grows with every message.
  • Prompt caching bills the repeated start of a prompt at a fraction of the normal input rate, roughly 90% off on most current models from OpenAI and Anthropic.
  • Batch processing costs 50% less at OpenAI and Anthropic if you can wait for results.

How an AI API bill is calculated

Every request has three parts that can show up on the bill:

Token typeWhat it isHow it's billed
InputYour instructions, the user's message, documents, conversation history, tool definitionsPer million input tokens
Cached inputThe part of the input that matches a recent request exactlyA reduced rate, plus a small write charge on some providers
OutputThe visible answer the model writesPer million output tokens, usually the highest rate
ReasoningInternal "thinking" before the answer, on reasoning modelsBilled as output tokens

OpenAI states that reasoning tokens "count toward output usage and are billed as output tokens" (OpenAI Help Center). Anthropic says the same about Claude's thinking tokens (Anthropic, Context windows). Depending on the problem, a reasoning model may generate "anywhere from a few hundred to tens of thousands of reasoning tokens" (OpenAI, Reasoning models). If you hit an output limit before the answer is finished, you can pay for input and reasoning and get no visible answer at all.

Price per token is also a weak way to compare models. OpenAI notes that "a lower price per million tokens does not necessarily produce a lower total cost," because models tokenize the same text differently and write different amounts. Anthropic, for example, says its Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text (Anthropic, Pricing). Compare models on cost per finished task, measured on your own examples.

Why context is the real cost driver

The context window is everything the model can see in one request: instructions, history, documents and its own answer. Anthropic calls it the model's "working memory" and warns that "more context isn't automatically better," because accuracy and recall degrade as token count grows (Anthropic, Context windows).

Each API request is independent and stateless, so a chat feature has to send the earlier messages again to keep the conversation going (OpenAI, Conversation state). Even when the provider stores the thread for you, "all previous input tokens for responses in the chain are billed as input tokens."

Here's a sample of how that adds up (illustrative numbers, not a real customer). A support chat has 2,000 tokens of instructions. Each turn adds a 300-token question and a 500-token answer. Over 10 turns the visible conversation is about 10,000 tokens, but the input you pay for across those 10 requests is about 59,000 tokens, because each request carries everything before it.

The usual sources of bloat:

  • Long system prompts with rules nobody has checked in months.
  • Full conversation history kept forever instead of trimmed or summarized.
  • Whole documents pasted in when a few relevant passages would do. Retrieval exists for this; see RAG explained.
  • Large tool definitions sent on every call, including tools the request will never use.
  • Agents that loop through many steps, each one resending the growing context.

Prompt caching: pay less for the part that repeats

Most production prompts start the same way every time: the same instructions, the same tool list, often the same reference documents. Prompt caching lets the provider reuse that processed prefix instead of handling it from scratch.

The details differ by provider:

OpenAIAnthropic (Claude)Google (Gemini)
How it turns onAutomaticYou mark what to cache, or turn on automatic cachingImplicit caching is on by default for Gemini 2.5 and newer
Minimum prefix1,024 tokens on GPT-5.6 and later1,024 tokens or more, by model2,048 to 4,096 tokens, by model
Write cost1.25x normal input (30-minute lifetime) on GPT-5.6 and later1.25x normal input (5-minute lifetime), 2x for 1 hourNo setup needed for implicit caching
Read cost0.1x on most of these models, 0.05x on some0.1x on most models, lower on some newer onesSavings passed on automatically on a hit

Sources: OpenAI, Prompt caching, Anthropic, Prompt caching and Pricing, Google, Context caching. These terms change often, so check the current page before you plan around them.

Whether caching actually pays off comes down to a few rules:

  • The prefix has to match exactly. Anthropic requires "100 percent identical prompt segments." A timestamp or user name near the top of the prompt breaks every hit, and you end up paying the write charge on every request with no reads.
  • Put stable content first. Instructions, tools and fixed documents go at the start; the user's question and anything that changes go at the end. Google gives the same advice for Gemini.
  • Prompts shorter than the threshold are simply not cached, and Anthropic returns no error when that happens. Check the usage fields in the response (for example cached_tokens or cache_read_input_tokens) to confirm you're getting hits.
  • Caches expire after minutes, so a feature used only a few times an hour may rarely hit a warm cache.
  • Cached tokens still take up space. Caching changes what you pay for those tokens, not whether they count toward the context window.

A worked example

This is a sample scenario with made-up round prices so the math is easy to follow. It doesn't describe any specific model or any of our projects.

An internal assistant handles 1,000 requests a day, about 30,000 a month. Each request has 6,000 tokens of fixed instructions and reference text, 2,000 tokens that change (the question plus retrieved passages), and a 500-token answer. Sample rates: $2 per million input tokens, $10 per million output tokens, cache writes at 1.25x and cache reads at 0.1x.

SetupMonthly inputMonthly outputTotal
No caching$480$150$630
Caching, 90% of requests hit the cache$197$150$347
Caching, plus answers capped near 300 tokens$197$90$287

Caching takes about 45% off this bill. After that, output is the biggest line, so the next win comes from shorter answers, a smaller model for simple requests, or both. Your real numbers depend on the model, the traffic pattern and how stable your prompts are, which is why we measure cost per request during a pilot before anyone commits to a budget. The pilot-to-production guide covers that step.

Other ways to control the monthly bill

  • Use the smallest model that passes your tests. OpenAI's cost guide lists "select a smaller model" first among its strategies (OpenAI, Cost optimization). Many apps route easy requests to a small model and send only hard ones to a large one.
  • Batch anything that can wait. OpenAI's Batch API offers a "50% cost discount" with results within 24 hours (OpenAI, Batch), and Anthropic's Batch API gives the same 50% off input and output (Anthropic, Pricing). Nightly document processing, tagging and report generation are good fits. Anthropic notes that batch and caching discounts can be combined.
  • Set the reasoning effort on purpose. Higher effort settings mean more reasoning tokens. OpenAI suggests using its highest setting only "when your evals show a clear benefit that justifies the extra latency and cost."
  • Cap output: set a maximum output length and ask for concise formats. Leave enough room for reasoning models to finish.
  • Trim history: keep the last few turns plus a short summary instead of the full thread. Compaction can help, though OpenAI notes it changes the prompt prefix and can reduce cache hits.
  • Set spend alerts and per-user limits. Unbounded usage is a cost risk and a security risk. Our guide on how much an AI agent costs to build and run covers limits and the other cost drivers.

FAQ

How do I estimate monthly AI running costs before building?

Take 20 to 50 real examples of the task, run them through the model you're considering, and record input, cached and output tokens per request from the usage data. Multiply by your expected monthly volume and add a margin for growth and retries. Token counters help with input, but output length only shows up when you actually run the task.

Is a bigger context window more expensive?

A larger window costs nothing by itself. Filling it does, because you pay for every token you send on every request. Some models also charge more for very long prompts. Send only what the task needs.

Does prompt caching change the answers?

No. OpenAI states that caching "does not change how the model generates output tokens." It changes what you pay for the repeated input and usually reduces latency.

Do I need my own servers to keep AI costs down?

Usually not. For most business features, a hosted API with caching, the right model size and sensible limits is cheaper and simpler than running models yourself. Self-hosting starts to make sense at high, steady volume or when data rules require it, and it brings its own hosting and maintenance costs.

Why did our AI bill jump when usage barely changed?

The usual causes are a prompt change that broke caching, conversations that keep growing, a switch to a reasoning model or higher effort, or an agent loop making more calls than expected. Check tokens per request and cache hit rates before you look at user numbers.

Get help keeping AI costs predictable

If you're adding AI to an existing product and want the running cost measured and under control from the start, we can help with prompt structure, caching, model choice, limits and cost monitoring. See our AI integration service, read how to add AI to your existing app without rebuilding it, or book a free consultation.

Sources

All checked November 2026. Provider prices and caching rules change often; check the current pages.