Book a call (opens Google Calendar)
Book a call (opens Google Calendar) hey@pronovahq.com

Your AI Bill Is a Workflow Problem, Not a Hardware Problem

By Izzy Aly, Co-Founder, Systems and Architecture

The leak is rarely the GPU. It's prompt bloat, retry storms, re-sent history, cache misses and outputs nobody reads. Where the workflow leaks, and what to fix first.

When the AI bill doubles, everyone looks at the hardware. The GPU type. The provider. Dedicated versus serverless. That's usually the wrong place to look.

I've watched this pattern for a year now. A team's inference spend spikes, someone opens a spreadsheet, and three weeks later they've migrated providers. Savings: almost nothing. Because the leak wasn't in the infrastructure. It was in the workflow sitting on top of it.

This post makes the case that for most teams, especially operators running agents against a CRM rather than racks of H100s, the bigger savings live one layer up. It maps where the money leaks in the workflow, and the controls that close each leak. And if you do run your own inference, it ends with the honest dedicated-vs-serverless math, redone against live pricing, because the numbers everyone quotes are stale.

The short version

  • Most AI budgets leak upstream of the GPU: bloated prompts, retry loops, conversation history re-sent on every turn, caches that never hit, and outputs nobody reads. No migration fixes that.
  • The human review gate that keeps an agent out of your production data is also a circuit breaker on your spend.
  • If you run your own GPUs: at RunPod's live pricing (retrieved October 28, 2026), dedicated only beats serverless above 66–86% utilization on Secure Cloud pods, or 51–64% on cheaper Community Cloud pods, depending on GPU. The old "30% rule" overbuys dedicated.

Where the money actually goes

Gartner's July 2024 prediction: at least 30% of generative AI projects get dropped after proof of concept by the end of 2025 (Gartner press release, 2024). Cost overruns were a named reason. The utilization data explains why. In ClearML's State of AI Infrastructure at Scale 2024 survey of 1,000 companies, 68% of respondents put their GPU utilization at 70% or below even at peak, and only 7% said they exceed 85% (ClearML, 2024). Modal puts the common number closer to 10–20% (Modal, 2026).

Read that again. Teams pay for capacity they use one hour in ten. Then they blame the price per hour.

But idle capacity is only the visible leak. The ones we find when we map a team's actual AI spend are quieter:

Prompt bloat. System prompts that ballooned to thousands of tokens because someone kept pasting in "just one more" instruction. A few hundred would do the same job. You pay the difference on every single call. Retrieval does the same thing at scale: to make sure the model has "enough" context, the app stuffs entire documents into the prompt "just in case". Sending a 50-page PDF to answer a question found on page 3 means you pay for far more tokens than you need. Even formatting costs money: stripping whitespace, indentation and newlines from code sent to a model cuts input tokens by about 24.5% on average, and the models tested do just as well without them (Pan et al., ICSE 2026).

Retry storms. An agent hits a malformed record, fails, and retries. And retries. RAG pipelines are especially good at this. A recursive search loop doesn't announce itself. It just shows up on the invoice. Retries are essential for resilience, but unbounded, immediate retries turn a transient provider blip into a self-inflicted DDoS. Use circuit breakers and exponential backoff: if errors spike, stop calling the provider. And give every agent loop a hard maximum number of iterations, so a confused agent can't cycle through "Thinking... Searching... Thinking..." until it times out. We cover how we cap agent budgets in a separate post.

Abandoned outputs. The weekly report an agent generates that nobody has opened since March. It runs. It costs money. It helps no one. Its cousins are the zombie experiments: that "v0.1" chatbot from last month's hackathon is still running in a staging environment. The engineer moved on, but the API key didn't. Put time-to-live policies on keys and strict budget caps on every non-production environment.

None of these get fixed by switching GPU providers. They get fixed by looking at the workflow.

Where the AI bill leaks: five steps above the GPU A left-to-right pipeline of five workflow steps, each with the leak it causes and the control that closes it. Prompt assembly: prompt bloat, whole documents stuffed into context, full history re-sent every turn and code formatting; control: trim the system prompt, retrieve only the top chunks, use rolling summaries and minify code, which cuts input tokens by about 24.5%. Model choice: a frontier model for every task; control: a small model first, escalating only when needed. Provider call: retry storms and cache misses; control: backoff and a circuit breaker, a cap on loop iterations and the static prompt prefix first. Output: output creep from wordier prompts and silent model updates; control: a max_tokens cap, a CI gate that blocks a 10% rise in cost per transaction and daily golden prompts. Who reads it: unread outputs and zombie experiments; control: a human review gate, time-to-live on keys and budget caps outside production. Underneath runs compute, which is rarely the biggest leak. 01 Prompt assembly LEAK Prompt bloat Stuffed docs Full history every turn Code spacing CONTROL Trim prompt Top chunks Summaries Minify code −24.5% input 02 Model choice LEAK Frontier model for every task CONTROL Small model first, escalate if needed 03 Provider call LEAK Retry storms Cache misses CONTROL Backoff and breaker Loop cap Static prefix first 04 Output LEAK Output creep Wordier prompts Silent model updates CONTROL max_tokens cap CI gate +10% Daily golden prompts 05 Who reads it LEAK Unread outputs Zombie experiments CONTROL Review gate Key TTLs Budget caps off-prod Compute (GPU, provider): rarely the biggest leak
Where the AI bill leaks: five steps above the GPU A left-to-right pipeline of five workflow steps, each with the leak it causes and the control that closes it. Prompt assembly: prompt bloat, whole documents stuffed into context, full history re-sent every turn and code formatting; control: trim the system prompt, retrieve only the top chunks, use rolling summaries and minify code, which cuts input tokens by about 24.5%. Model choice: a frontier model for every task; control: a small model first, escalating only when needed. Provider call: retry storms and cache misses; control: backoff and a circuit breaker, a cap on loop iterations and the static prompt prefix first. Output: output creep from wordier prompts and silent model updates; control: a max_tokens cap, a CI gate that blocks a 10% rise in cost per transaction and daily golden prompts. Who reads it: unread outputs and zombie experiments; control: a human review gate, time-to-live on keys and budget caps outside production. Underneath runs compute, which is rarely the biggest leak. STEP LEAK CONTROL 01 Prompt Bloat, history, code spacing Trim, summarise, minify −24.5% 02 Model Frontier model by default Small model, then escalate 03 Provider Retry storms, cache misses Backoff, loop cap, static first 04 Output Output creep, model updates max_tokens, CI gate +10% 05 Readers Unread outputs, zombie tests Review gate, TTLs, caps Compute: rarely the biggest leak
Fig. 1 — Where the bill leaks: five workflow steps above the GPU, each with its leak and the control that closes it. Source: Pan et al., ICSE 2026 for the 24.5% figure.

Three leaks that grow with usage

The leaks above are waste you can point at. These three are harder to see, because they get worse as the product succeeds.

The conversation-history tax

LLMs are stateless. Most developers implement memory by re-sending the conversation history with every new user prompt.

  • Turn 1: User says "Hi." (Cost: negligible)
  • Turn 10: System sends "Hi" + 9 previous turns + system prompt + new question.

Your costs don't scale linearly; they scale cumulatively. This creates a specific risk for flat-rate subscription models: your most engaged power users, the ones you want to retain, can become your least profitable customers because of their token consumption.

The conversation-history tax, and how a rolling summary flattens it Two panels of stacked blocks, one stack per conversation turn from 1 to 10. Left, re-send everything: each turn sends the system prompt, every previous turn and the new question, so turn 1 is the system prompt plus "Hi" and turn 10 is the system prompt plus nine previous turns plus the new question; the stack grows every turn and cost grows cumulatively. Right, rolling summary: each turn sends the system prompt, one fixed-size summary of older turns and the new question, so the stack stays the same height. Conceptual, not to scale. Re-send everything cost climbs every turn Rolling summary cost stays flat 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 TURN System prompt Previous turns Summary New question
The conversation-history tax, and how a rolling summary flattens it Two panels of stacked blocks, one stack per conversation turn from 1 to 10. Left, re-send everything: each turn sends the system prompt, every previous turn and the new question, so turn 1 is the system prompt plus "Hi" and turn 10 is the system prompt plus nine previous turns plus the new question; the stack grows every turn and cost grows cumulatively. Right, rolling summary: each turn sends the system prompt, one fixed-size summary of older turns and the new question, so the stack stays the same height. Conceptual, not to scale. Re-send everything climbs every turn Rolling summary stays flat 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 System Past turns Summary New question
Fig. 2 — Re-sending the full history makes every turn cost more than the last. A rolling summary keeps the context a fixed size. Conceptual, not to scale.

The fix is to stop sending everything:

  • Rolling summaries: a background process summarizes older conversation turns into a concise paragraph, keeping the context window a fixed size rather than growing indefinitely.
  • Selective retrieval: instead of dumping every possible tool definition into the prompt, a retrieval step loads only the tools needed for that turn.

The caching paradox

There are two kinds of caching, and both are easy to get wrong.

The first is caching answers. If 1,000 users ask "How do I reset my password?", a system that treats every request as new generates the answer 1,000 times. Semantic caching returns a stored answer when a new question is close enough to a cached one, so you pay for the lookup, not for new tokens.

The second is prompt caching: the provider stores the processed state of a prompt prefix. It can cut the bill a lot, but vendors price it differently. OpenAI caches automatically; on GPT-5.6 and later it charges 1.25× to write to the default 30-minute cache and gives a 90% discount on reads, 95% on GPT-6.1 Sol and Terra (OpenAI). Anthropic charges a premium to write to the cache (+25% for the default five-minute cache) and gives a 90% discount when you read from it, 95% on Claude Opus 5.5 and Sonnet 5.5 (Anthropic).

Suppose your engineers enable caching on a prompt that changes frequently, say one with a timestamp or a user-specific variable before the cached section. You never hit the cache. You pay the write premium repeatedly and never receive the read discount. Place static instructions (system prompts, few-shot examples) at the very top and dynamic user data at the bottom, and watch the hit rate: below about 20%, caching may be adding to your bill.

Output creep

Output tokens cost several times as much as input tokens, so anything that makes responses longer moves the bill fast. Two things do that without anyone deciding to.

A prompt change that makes the AI "more polite" might double the length of its responses. Engineers change a prompt, test it on three examples, decide it "feels better," and push to production. Without cost regression testing, you won't know you've doubled your bill until the invoice arrives.

And model providers update their backend models. Sometimes an update changes how the model interprets "verbosity": your prompt still says "Be concise", and the average response still gets longer. Your tests break and costs spike, but nobody changed a line of code.

The fix:

  • Output token limits: hard cap max_tokens in your API calls.
  • A cost gate in CI/CD: block a deploy if the projected cost per transaction rises by more than 10%.
  • Golden prompts: a staging environment that runs a fixed set of prompts against the model daily, to detect drift in verbosity or reasoning cost.

The gate that protects your data also caps your bill

Every workflow we build ships with a human review gate: the agent drafts, a person approves, then it executes. We do that for governance: nothing touches a production system without a human saying yes.

Here's the side effect we didn't advertise enough: a gated workflow can't run away with your budget. A retry loop that needs approval every cycle dies at cycle one. An agent that drafts before it sends can't burn a weekend generating output nobody asked for.

This pattern is built into our open-source HubSpot plugin: preview, approve, undo, audit. The same gate that keeps a coding agent from corrupting your CRM is a circuit breaker on your token spend. One mechanism, two jobs.

If you're subject to the EU AI Act, that audit trail isn't optional anyway: Articles 12 and 19 require logging for high-risk systems, and since the Digital Omnibus entered into force on 27 July 2026 (Regulation (EU) 2026/1744), the obligations for Annex III high-risk systems apply from 2 December 2027. Fines reach €35M or 7% of global turnover for the most serious infringements (prohibited practices); breaches of the high-risk obligations sit in the €15M or 3% tier (Article 99). But the more immediate risk is commercial: European enterprise buyers block AI tools that lack clear data residency and audit trails. If you can't prove where the data lives and how decisions are logged, you lose the deal. Don't bolt it on later. Log inputs and outputs from day one.

Routing beats GPU shopping

One more finding that supports the "look upstream" argument. Google's GKE team published Vertex AI production numbers in February 2026: putting a smarter gateway in front of their model servers (load-aware and cache-aware routing, a software-layer change) cut P95 time-to-first-token by 52% on bursty chat workloads and doubled their prefix cache hit rate from 35% to 70% (Google Cloud, 2026).

No hardware swap in the write-up. Half the tail latency. The layer between your users and your compute, the orchestration layer, moved the needle more than any hardware swap would have. That's the pattern, whether you're routing LLM traffic at Google scale or deciding which of your 3+ SaaS tools an agent should touch.

The same layer is where three cheaper fixes live:

Stop taking a Ferrari for groceries. Too many workloads quietly default to the most powerful model available, often hardcoded into an environment variable, because big models are forgiving and engineers ship faster with them. Using a frontier model for a simple task, like extracting a date from an email, can cost 4x–40x what a smaller model would: on Anthropic's current price list, Claude Opus 5.5 costs 4 times as much per token as Haiku 4.5 and 40 times as much as Haiku 5.5 (Anthropic). Cascade instead: a small, fast model takes the simple requests, and the expensive reasoning model gets a request only when the small one has low confidence or the query is classified as complex. This does add architectural complexity; you have to build and test the routing logic.

Close the open loop. When services call OpenAI, Anthropic or Google directly with keys from local config files, there is no central traffic control. You can't see aggregate volume, you can't throttle a specific service, and you can't stop a runaway loop in a dev environment until the credit card limit hits. Route all traffic through a gateway that holds the keys, logs requests and enforces rate limits.

Give every cost line an owner. When the whole platform runs on one API key belonging to "the platform team", Team A builds an inefficient feature and someone else pays the bill. Use virtual keys or metadata tags (feature_id, team_id) so each line of spend belongs to a team and a feature. Then show engineers the price of their prompts in staging. Engineering is rewarded for velocity and uptime, rarely for unit cost; when a developer sees "this prompt costs €0.15 per run", they optimize it before it hits production.

If you run your own GPUs: dedicated vs. serverless

If you do run your own inference, the question's real. The industry rule of thumb said dedicated wins above 30% utilization. Against current pricing, that rule's wrong. By a lot.

Break-even utilization = dedicated hourly rate ÷ serverless hourly rate. Below that number, serverless is cheaper. Above it, dedicated wins.

At RunPod's live on-demand pricing (retrieved October 28, 2026), an L4, A100 80GB and H100 80GB cost $0.59, $1.79 and $3.99/hr for pods against $0.69, $2.72 and $4.79/hr serverless: break-even at about 86%, 66% and 83%. RunPod's cheaper Community Cloud pods ($0.44, $1.39 and $2.69/hr) bring it down to about 64%, 51% and 56%.

Break-even utilization, dedicated vs serverless GPUs at RunPod pricing Horizontal bars on a 0 to 100% utilization axis show where a dedicated RunPod pod becomes cheaper than serverless, retrieved 28 October 2026. Secure Cloud pods: L4 24GB about 86%, A100 80GB about 66%, H100 80GB about 83%. Community Cloud pods: L4 about 64%, A100 about 51%, H100 about 56%. A dashed line marks the old 30% rule of thumb, and a shaded band marks the 10 to 20% utilization Modal says is common. Every break-even sits far to the right of both. UTILIZATION WHERE DEDICATED BEATS SERVERLESS Secure Cloud pod Community Cloud pod 10–20% typical old 30% rule L4 24GB 86% 64% A100 80GB 66% 51% H100 80GB 83% 56% 0% 25% 50% 75% 100% Break-even = dedicated hourly rate ÷ serverless hourly rate. Source: RunPod on-demand pricing, retrieved 28 October 2026; typical utilization from Modal.
Break-even utilization, dedicated vs serverless GPUs at RunPod pricing Horizontal bars on a 0 to 100% utilization axis show where a dedicated RunPod pod becomes cheaper than serverless, retrieved 28 October 2026. Secure Cloud pods: L4 24GB about 86%, A100 80GB about 66%, H100 80GB about 83%. Community Cloud pods: L4 about 64%, A100 about 51%, H100 about 56%. A dashed line marks the old 30% rule of thumb, and a shaded band marks the 10 to 20% utilization Modal says is common. Every break-even sits far to the right of both. WHERE DEDICATED WINS Secure Cloud Community Cloud typical 10–20% old rule 30% L4 86% 64% A100 66% 51% H100 83% 56% 0% 50% 100% Break-even = dedicated ÷ serverless rate. Source: RunPod pricing, 28 Oct 2026; Modal.
Fig. 3 — Dedicated only beats serverless far above the old 30% rule, and further still above the 10–20% utilization most teams run at. Source: RunPod pricing, retrieved 28 October 2026; typical utilization from Modal.

You'd need your H100 busy roughly 20 hours a day before owning it beats renting it by the second. Most teams aren't close. Cold starts no longer rule serverless out: Modal cut inference spin-up from roughly 2,000 seconds to about 50 (Modal, 2026).

  • Steady traffic and you've measured utilization above your break-even? Dedicated.
  • Spiky or sporadic traffic? Serverless, or a hybrid: a small dedicated floor, serverless for peaks.
  • Sub-100ms SLA or strict data residency? Dedicated, in your own VPC. No debate.

Where this still leaks

Full disclosure, in two parts.

First: pricing tables go stale in weeks. The break-evens above were true at one provider when we retrieved them. Recompute them with your own quotes before you take them into a budget meeting. The formula is one division.

Second: we're an implementation shop, not a GPU infrastructure consultancy. If you're running a large fleet of dedicated H100s, this post's the appetizer and your infra team owns the meal. Our lane is the layer above: the workflows, gates, and orchestration that decide what hits your compute in the first place.

Common questions

Is serverless GPU cheaper than dedicated?

Below roughly 51–86% utilization, depending on GPU and tier, yes. At RunPod's October 2026 pricing, an A100 costs $1.79/hr dedicated (Secure Cloud) versus $2.72/hr serverless, so dedicated only wins if it's busy more than ~66% of the time. Modal puts typical utilization at 10–20%, which makes serverless the cheaper default.

We don't run GPUs; we pay per token. Does any of this apply?

All of it. Capacity-versus-usage is the same call as committed-spend contracts versus pay-as-you-go API pricing. And the workflow leaks (prompt bloat, retry loops, re-sent history, unread outputs) inflate a token bill exactly the way they inflate a GPU bill. Audit the workflow first.

What should I measure before deciding anything?

Two numbers: your actual utilization (or token spend per workflow, if you're API-based) and your cost per successful outcome, not per token. Thirty days of data beats any rule of thumb, including the ones in this post.

Run your own numbers. If you'd rather map the workflow with someone who has done it before, book a scoping call.

Topics
Author
Izzy Aly · Co-Founder, Systems and Architecture
Share

Keep reading

All posts →