LLM cost optimization is less about choosing the cheapest model and more about controlling where, when, and how many tokens your application sends. This guide provides a repeatable token-budget worksheet, model-routing rules, prompt-compression techniques, semantic-caching patterns, batch-processing guidance, and review checklists for reducing AI API costs without allowing quality or reliability to drift.
Overview
AI API bills usually come from a combination of input tokens, output tokens, request volume, model selection, retries, and long-running workflows. A useful cost-control plan measures each factor separately instead of treating the total invoice as a single problem.
The basic estimate is:
Estimated cost = requests × ((input tokens × input rate) + (output tokens × output rate))
Use the provider’s current pricing units in place of “rate.” Some providers price input and output differently, offer separate treatment for cached input, or provide batch-processing options with their own rules. Do not copy a rate from an old spreadsheet without checking the current model documentation.
Cost reduction should also include operational effects. A smaller model that needs more retries may cost more than a larger model that succeeds on the first attempt. Likewise, an aggressive context reduction that lowers answer quality can increase support work, human review, or downstream API calls. The target is not minimum token volume; it is the lowest total cost that still meets your quality, latency, safety, and availability requirements.
For features that use retrieval, agents, or sensitive data, pair cost analysis with engineering controls. A production RAG tutorial can help isolate retrieval costs from generation costs, while the guide to AI guardrails for production apps covers checks that should not be removed simply to save tokens.
How to estimate
Start with observed traffic rather than an idealized request. Export a representative sample from logs and calculate the median and high-percentile input and output token counts. If you do not have production data, use a clearly labeled planning assumption and replace it after launch.
- Define the unit of work. Decide whether you are estimating one chat turn, one document, one agent task, or one completed workflow. A workflow may contain several model calls.
- Count calls per unit. Include classification, retrieval query rewriting, tool selection, generation, validation, and repair calls. Retries should be modeled separately rather than hidden in an average.
- Separate input and output tokens. Record system instructions, user content, retrieved passages, tool results, conversation history, and generated output as different components.
- Apply the current rates and billing rules. Verify the model’s input, output, cached-input, batch, minimum, and rounding rules before publishing a budget or approving a migration.
- Multiply by traffic scenarios. Build low, expected, and high-volume cases. Include a temporary spike scenario if traffic is seasonal or a new feature may change usage.
- Compare cost per successful outcome. Track completed tasks, accepted answers, or resolved cases—not only requests. Quality failures and retries belong in the estimate.
A practical worksheet can use these columns:
- Feature or workflow name
- Requests per day or month
- Model and version
- Calls per request
- Average input tokens per call
- Average output tokens per call
- Retry rate and average retry count
- Cache hit rate, if applicable
- Batch-eligible volume
- Estimated cost and cost per successful outcome
For a more reliable forecast, maintain two views: a request-level estimate for debugging and a monthly estimate for finance. The request view reveals which prompt or workflow is expensive; the monthly view reveals whether a small inefficiency becomes material at scale.
Inputs and assumptions
Prompt and context size
Long system prompts, repeated instructions, conversation history, retrieved documents, and verbose tool schemas can dominate input usage. Compress context by removing duplicated rules, limiting retrieved passages to evidence that can affect the answer, and summarizing older conversation turns. Preserve requirements, definitions, constraints, and examples that materially influence behavior.
Prompt compression should be evaluated, not assumed. Keep a test set of representative inputs and compare task success, factuality, refusal behavior, structured-output validity, latency, and tokens. A prompt testing framework makes this comparison easier to repeat when prompts or models change.
Model routing
Use a routing policy based on task difficulty and risk. A simple policy can look like this:
| Workload | Default route | Escalate when |
|---|---|---|
| Classification, extraction, and routing | Lower-cost model with constrained output | Confidence is low or schema validation fails |
| Routine drafting and summarization | Efficient general-purpose model | Evaluation scores or user feedback fall below the threshold |
| Complex reasoning or ambiguous requests | More capable model | Only when the task requires it |
| High-risk or customer-facing decisions | Approved model and guarded workflow | Human review or a defined fallback is required |
Route by attributes you can measure: input length, task type, language, confidence, tool requirements, and failure history. Avoid routing solely on a vague “easy versus hard” label. Log the route and outcome so the policy can be tuned against real results.
Prompt caching and semantic caching
Prompt caching is useful when a request repeatedly includes an identical or stable prefix, such as a system instruction set, policy document, or large tool definition. Put stable content before variable content where the provider’s caching rules support that pattern. Confirm eligibility, minimum sizes, expiration behavior, and billing treatment in the current provider documentation.
Semantic caching is an application-level cache. Before calling a model, normalize the user request, create or retrieve an embedding, and search for a previous request that is sufficiently similar. Return a cached answer only when the task, permissions, data freshness, and required response format are compatible. For dynamic or sensitive data, attach a short validity period or skip semantic caching entirely. Never allow a cache key to cross tenant or authorization boundaries.
Batch processing
Batch API processing can fit offline work such as document enrichment, nightly classification, evaluation runs, and non-urgent summaries. Build the workflow so jobs are idempotent, can be retried by item, and record the model, prompt version, input identifier, and output status. Batch is not a substitute for a synchronous path when users are waiting for an immediate answer.
Check provider-specific batch limits, completion windows, file formats, error handling, retention behavior, and pricing before relying on it. Treat those inputs as changeable assumptions in your worksheet.
Worked examples
Example 1: Reducing a support-summary workflow
Assume a support application processes 10,000 tickets per month. Each ticket currently makes one call with 4,000 input tokens and 700 output tokens. The estimate is therefore 40 million input tokens and 7 million output tokens per month, before retries.
The team identifies three changes: a shorter system prompt removes 500 input tokens per call, retrieval is limited to relevant evidence and removes another 1,000 tokens, and a smaller model handles straightforward summaries. The revised input volume becomes 25 million tokens per month. The team should calculate the new cost using the selected model’s current rates, then compare summary acceptance and escalation rates with the baseline. The savings are real only if quality remains within the agreed threshold.
Example 2: Adding a semantic cache
Assume a product answers recurring internal policy questions. The team observes that many requests are substantially similar, but not textually identical. It introduces a tenant-scoped semantic cache with a conservative similarity threshold and a short freshness window. The worksheet now includes cache lookups, cache-hit rate, embedding cost, invalidation events, and model calls avoided.
For example, if 10,000 monthly requests produce 2,000 accepted cache hits, the generation estimate applies to 8,000 requests rather than 10,000. That does not automatically equal a 20% total saving: embedding calls, cache storage, misses, invalidations, and occasional revalidation still have costs. Measure total workflow cost and answer correctness, not the hit rate in isolation.
Example 3: Moving evaluation work to batches
A team runs 50,000 prompt-evaluation cases after every major change. These cases do not need interactive latency. The team packages each case with a stable identifier, prompt version, expected checks, and retry status, then runs them through an approved batch path. The cost worksheet separates batch processing from production traffic and records completion time, failed items, and rerun volume. This prevents evaluation traffic from obscuring the cost of customer-facing requests.
When to recalculate
Recalculate the estimate whenever a provider changes pricing, tokenization, context-window rules, caching treatment, batch limits, or model capabilities. Also recalculate after a model migration, prompt revision, retrieval change, new tool integration, traffic increase, or unexpected billing event. These are not rare maintenance details; they are normal inputs to production AI engineering.
Pre-launch checklist
- Measure representative input and output tokens for every model call.
- Include retries, fallbacks, agent loops, tool calls, and validation calls.
- Document pricing and operational assumptions with the date checked.
- Set quality, latency, safety, and cost thresholds for each route.
- Test compressed prompts and smaller models against a fixed evaluation set.
- Define cache scope, freshness, invalidation, and authorization rules.
- Confirm whether offline work is suitable for batch processing.
- Add per-feature and per-tenant usage monitoring where appropriate.
Monthly billing review
- Compare forecast tokens with actual input, output, cached, and batch usage.
- Find workflows with rising average context or output length.
- Review retry rates, fallback rates, cache hits, and cost per successful outcome.
- Sample quality after every routing or compression change.
- Check for abandoned experiments, duplicate requests, and runaway agent loops.
- Refresh provider inputs before approving the next budget.
Keep the worksheet versioned alongside prompts and application code. A cost estimate is most useful when engineers can explain why it changed, reproduce its assumptions, and connect the change to a measurable product outcome.
