Skip to content
Gradient Notes logoGradient Notes
Dense bundle of network cables running into a server rack
Infrastructure

The hidden cost of tokens

Per-token prices keep falling, yet our bill kept rising. Where the tokens were really going, and the five changes that cut spend by 61%.

Daniel Reyes2 min read

Our model provider cut prices twice last year. Our monthly bill went up anyway. When we finally broke spend down by feature and by request, the culprit was not volume. It was waste: tokens we paid for that did nothing for the user.

Measure per feature, not per account

The first step was boring and essential: tag every model call with the feature that made it, then build a dashboard of tokens in, tokens out and cost per feature per day. Within an hour of shipping the dashboard we found that a background "summarise this ticket" job, which nobody looked at, accounted for 34% of spend.

Where the waste hid

Bloated system prompts

Our main assistant prompt had grown to 3,100 tokens of accumulated instructions, examples and edge-case patches. It was sent on every turn.

Resending history

Multi-turn chats resent the full conversation every time. Long sessions cost twenty times more than short ones for little benefit.

Unused background jobs

Jobs that summarised, classified or tagged content "in case it was useful" ran on every update, even when nothing read the output.

The five changes

  1. Trimmed the system prompt. We deleted instructions the eval set showed made no difference. 3,100 tokens became 900, with no measurable quality loss.
  2. Turned on prompt caching. Stable prefixes (system prompt, tool definitions) are cached by most providers at a steep discount. Ordering the prompt so the stable part comes first was a one-line change.
  3. Summarised long conversations. After eight turns we replace older messages with a running summary.
  4. Routed easy requests to a small model. A cheap classifier decides whether a request needs the large model. Seventy percent do not.
  5. Made background jobs lazy. Summaries are now generated the first time someone opens the view, then cached.
“Cheaper tokens are a discount on waste. Fewer tokens is the actual saving.”
— Daniel Reyes

Results

Over six weeks, monthly spend fell 61% while traffic grew 18%. P95 latency improved too, because shorter prompts are faster prompts. None of these changes required a new model; all of them required knowing where the tokens went.

Keep reading