Book a call (opens Google Calendar)
Book a call (opens Google Calendar) hey@pronovahq.com

Fine-Tuning vs. RAG: The Real Cost-Governance Playbook for Growing Teams

By Izzy Aly, Co-Founder, Systems and Architecture

79% of enterprises overspent on AI in the past year (DoiT/Sapio, 2026). Here's when fine-tuning, RAG, prompt caching or the missing governance layer is your real cost problem.

It usually starts with a Slack message: "Why did our AI bill triple last month?" This post answers that question directly: whether fine-tuning, RAG, or the governance layer around both is your real cost driver, and how to tell which one before you spend on any of them.

If you're the person who got that message, you're not alone, and you're probably not the problem. 79% of enterprises experienced AI cost overruns in the past 12 months, according to a February 2026 survey of 500 finance leaders by Sapio Research, commissioned by DoiT (DoiT, "Why 79% of Enterprises Overspent on AI in 2026"). The counterintuitive part: the organisations that rate their FinOps practice as most mature reported the highest overruns, at 89%, because they're the only ones instrumented well enough to catch it. The cost overrun is rarely the model. It's the missing governance layer around it.

Key Takeaways

  • Fine-tuning buys consistent behaviour; RAG buys current knowledge. Picking one instead of matching each to its job is what causes cost surprises.
  • For facts that fit in the prompt, prompt caching cuts the input bill without a training run, and OpenAI is winding down its self-serve fine-tuning platform anyway.
  • 79% of enterprises overspent on AI in the past year, worst among the best-instrumented teams, because they're the only ones who could see it (DoiT/Sapio, 2026).
  • EU AI Act high-risk fines reach €15M or 3% of global turnover; auditability is a design decision, not a bolt-on.

Why do AI bills blow up in growing companies?

The pattern in the DoiT/Sapio data is structural, not technical: accountability for AI spend splits almost evenly between technology (55%) and finance (53%) leadership, with no clear single owner at the operational level, and only 15% of finance leaders can calculate AI ROI without significant bottlenecks (DoiT, 2026). Nobody's job is to notice the bill until it's already a board question.

Agentic workflows make this worse, not better. Gartner forecasts that over 40% of agentic AI projects will be cancelled by the end of 2027, "due to escalating costs, unclear business value or inadequate risk controls" (Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027", Jun 2025). The models aren't failing. The business management around them is: no cost owner, no approval gate, no baseline measurement before the rollout started.

Part of that business-management failure is vendor-driven. Gartner also estimates that only about 130 of the thousands of agentic AI vendors on the market today build systems with real agentic capability, the rest are existing chatbots and RPA tools rebranded to ride the hype, a practice Gartner calls "agent washing" (same source). Buying the rebrand instead of the capability is its own line item in next quarter's overrun.

This is the moment we build for. Not "should we fine-tune or use RAG" in the abstract, but: your team is already running AI-powered workflows, the bill just moved, and you need to know exactly why before you can defend the budget or fix the leak.

What do fine-tuning and RAG actually buy you?

Before pricing anything, it's worth being precise about what you're buying, because the two are frequently pitted against each other when they solve different problems.

Fine-tuning takes a pre-trained model and trains it further on your data, changing its weights so it internalises your terminology, tone, and formats. You're buying behaviour: a model that already sounds like you and outputs in the shape you need, every time.

RAG (Retrieval-Augmented Generation) keeps the model generic and gives it access to your data through a retrieval layer, typically a vector database. You're buying knowledge that stays current: the model reasons over documents it's handed at query time instead of trying to remember everything.

Neither is "the AI." Both are components you'd choose, combine, or skip depending on what the workflow actually needs.

What are we actually optimising for?

We evaluate the fine-tuning vs. RAG trade-off against three things, because these are the three things that turn into board questions:

  1. Capital efficiency. Not just inference cost, but the ongoing labour of keeping data and evaluations current, what we've started calling the CFO Test: can you show clear unit economics, predictable spend, and an explainable driver for every dollar.
  2. Determinism and auditability. Can you show why the system gave that specific answer? This is the EU AI Act question, not a nice-to-have.
  3. Change tolerance. What happens the week your knowledge base changes, your pricing changes, or your policy changes? Some architectures absorb that with a database write. Others need a retraining cycle.

Fine-tuning: strong for behaviour, expensive for facts

Fine-tuning earns its keep on relatively stable, well-defined tasks: enforcing brand voice, strict output formats like JSON or SQL, and narrow tasks where a small tuned model can outperform a much larger general-purpose one, provided you're running it efficiently.

It struggles the moment the underlying facts move. If you fine-tune a model on this quarter's pricing or last year's HR policy, the model doesn't know it's wrong. It just keeps confidently giving you last year's answer, and there's no clean way to trace that answer back to a source document, because it's baked into the weights, not stored as a citable fact. Naively retraining to fix it risks catastrophic forgetting: degrading behaviour the model already had right. The same weights make personal data hard to unlearn or audit once it's in there.

The bill has two hidden lines. The first is iteration: a production-ready fine-tuned model rarely emerges from a single training run, so you're paying for an R&D cycle, not one run. The second is the inference premium. Fine-tuning is often sold as a cost-saver ("shorter prompts!"), but the unit economics change. On OpenAI's price list, a fine-tuned GPT-4.1 costs $3.00 per million input tokens against $2.00 for the base model, and a fine-tuned GPT-4.1 mini costs $0.80 against $0.40, on top of $25 and $5 per million training tokens (OpenAI pricing). You pay that premium on every interaction, for as long as the model runs, and GPT-4.1 and GPT-4.1 mini, fine-tuned versions included, leave the API on 14 January 2027 (OpenAI deprecations).

The fine-tuning premium on every input token Paired bar chart of OpenAI input prices per million tokens. GPT-4.1: base model $2.00, fine-tuned $3.00. GPT-4.1 mini: base model $0.40, fine-tuned $0.80. Training adds $25 per million training tokens for GPT-4.1 and $5 for GPT-4.1 mini. OpenAI is winding down self-serve fine-tuning: no new jobs from 6 January 2027, and GPT-4.1 fine-tunes stop serving on 14 January 2027. INPUT PRICE PER MILLION TOKENS, OPENAI GPT-4.1 Base $2.00 Fine-tuned $3.00 GPT-4.1 mini Base $0.40 Fine-tuned $0.80 Training adds $25 (GPT-4.1) and $5 (GPT-4.1 mini) per million training tokens. OpenAI is winding down self-serve fine-tuning. No new jobs from 6 January 2027. GPT-4.1 fine-tunes stop serving 14 January 2027.
The fine-tuning premium on every input token Paired bar chart of OpenAI input prices per million tokens. GPT-4.1: base model $2.00, fine-tuned $3.00. GPT-4.1 mini: base model $0.40, fine-tuned $0.80. Training adds $25 per million training tokens for GPT-4.1 and $5 for GPT-4.1 mini. OpenAI is winding down self-serve fine-tuning: no new jobs from 6 January 2027, and GPT-4.1 models, fine-tunes included, retire on 14 January 2027. INPUT PRICE PER 1M TOKENS, OPENAI GPT-4.1 Base $2.00 Fine-tuned $3.00 GPT-4.1 mini Base $0.40 Fine-tuned $0.80 Training: $25 and $5 per 1M training tokens OpenAI is winding down fine-tuning No new jobs from 6 Jan 2027 GPT-4.1 models retire 14 Jan 2027
Fig. 1 — A fine-tuned model costs more per input token than its base model, on every call, on top of the training run. Source: OpenAI pricing; OpenAI deprecations.

The vendor ground has moved too. OpenAI is winding down its self-serve fine-tuning platform: since 7 May 2026, organisations that never ran fine-tuning can't start, and from 6 January 2027 active existing customers can't create new jobs. Inference on existing fine-tunes ends when the base model is retired, for example on 14 January 2027 for GPT-4.1 and GPT-4.1 mini (OpenAI deprecations). Anthropic's fine-tuning route was Claude 3 Haiku on Amazon Bedrock, and that model reached end of life in most Bedrock regions, including US West (Oregon), on 10 September 2026 (AWS). Google still offers supervised tuning on its Gemini Enterprise Agent Platform (formerly Vertex AI), for Gemini 2.5 Pro and for Flash and Flash-Lite models up to Gemini 3.5 (Google Cloud).

Use it for: brand voice, formatting, narrow and stable tasks. Don't rely on it for: anything that changes on a schedule shorter than your retraining cycle.

RAG: strong for facts, adds moving parts

RAG shifts most of your spend to usage, tokens and retrieval, rather than training runs. Updating the knowledge base is a database write, not a training job, which is why it tends to win when your underlying facts move faster than a retraining cycle. It's also the architecture that gives you something to point to when someone asks "why did the model say that": a retrieved document, with a citation, that you can show an auditor.

The trade-off is real: you're feeding documents into every prompt, which adds token cost that compounds on long conversations, and you're now maintaining a vector database and retrieval logic as permanent infrastructure, not a one-time build.

Use it for: enterprise search, support bots, policy assistants, anything where the facts change and you need to prove where an answer came from. Lower priority for: open-ended creative work where external grounding doesn't add much.

Where does prompt caching fit?

For facts that fit in the prompt, a third option needs no training run and no vector database. Prompt caching exploits the Key-Value (KV) cache in transformer architectures: instead of re-computing the attention states for your long system prompt every time a user asks a question, the API provider stores that computed state and bills the reused tokens at a discount.

On OpenAI, caching is on by default for supported models, and cached input on GPT-5.4 costs $0.25 per million tokens against $2.50 uncached (OpenAI pricing). Take 5,000 tokens of context across 1M queries a month: 5B input tokens cost $12,500 uncached and $1,250 if every one of them is read from cache. On Anthropic, you turn it on with a cache_control field. The cache lives 5 minutes by default or 1 hour at extra cost, and cache reads cost 0.1 times the base input price on most models, 0.05 times on Opus 5.5 and Sonnet 5.5, and 0.025 times on Fable 5.1 and Mythos 5.1 (Anthropic).

The Achilles' heel of caching is invalidation. If you change even one token in the prefix, the cache for that block is invalidated. Place stable content (company mission) first, moderately stable content (product catalogue) second, and dynamic content (the user's query) last. Timestamps or user IDs before the cache breakpoint kill your hit rate. Content that changes faster than the cache lives doesn't belong in a cached prefix either; use standard RAG.

The rule of thumb: if your documents fit in the prompt, cache them whole. Once they don't, use RAG.

Fine-tune, cache or retrieve: one question decides Decision flow. Start with what the model needs. If it is how the model acts (behaviour such as tone, format or strict JSON), fine-tune, where the vendor still offers it. If it is what the model knows (information such as prices, policies and documents), ask whether the documents fit in the prompt. Yes: use prompt caching with stable content first and dynamic content last. No: use RAG, a retrieval layer with citations. Example: 5,000 tokens of context across 1 million queries is 5 billion input tokens a month on GPT-5.4, which costs $12,500 at $2.50 per million uncached and $1,250 at $0.25 per million cached. FINE-TUNE, CACHE OR RETRIEVE What does the model need? How it acts tone, format, JSON What it knows prices, policies, docs Fine-tune where the vendor still offers it Fits in the prompt? Yes: prompt caching stable first, dynamic last No: RAG retrieval layer, citations 5,000 tokens × 1M queries = 5B input tokens a month on GPT-5.4 $2.50/M uncached: $12,500. $0.25/M cached: $1,250.
Fine-tune, cache or retrieve: one question decides Decision flow. Start with what the model needs. If it is how the model acts (behaviour such as tone, format or strict JSON), fine-tune, where the vendor still offers it. If it is what the model knows (information such as prices, policies and documents), ask whether the documents fit in the prompt. Yes: use prompt caching with stable content first and dynamic content last. No: use RAG, a retrieval layer with citations. Example: 5,000 tokens of context across 1 million queries is 5 billion input tokens a month on GPT-5.4, which costs $12,500 at $2.50 per million uncached and $1,250 at $0.25 per million cached. FINE-TUNE, CACHE OR RETRIEVE What does the model need? Fine-tune behaviour tone, format, JSON; where still offered Prompt caching information, fits stable first, dynamic last RAG information, too big retrieval layer, citations 5B input tokens a month on GPT-5.4 $12,500 uncached, $1,250 cached
Fig. 2 — Behaviour goes to fine-tuning. Information goes into a cached prompt if it fits and into RAG if it doesn't. Source: OpenAI pricing.

Head-to-head: fine-tuning vs. RAG

Fine-tuning wins on tone and format consistency but needs a retraining cycle to absorb a knowledge change. RAG updates with a database write and produces the document-level citations an audit needs, at the cost of higher ongoing token and retrieval spend. Here's the full comparison:

DimensionFine-TuningRAG
Knowledge freshnessRetraining cycleDatabase update
AuditabilityWeights are opaque; relies on loggingDocument-level citations, when designed for it
Tone & format consistencyStrongDepends on prompting discipline
Ongoing cost driverRetraining + evaluationTokens + retrieval infra
Best fitStable behaviour, brand voiceChanging facts, compliance trail

Neither column wins outright. Very high, stable-volume workloads can favour fine-tuning on unit economics alone. Everything else, which is most real teams, favours RAG or a hybrid.

Should you choose fine-tuning or RAG, or combine them?

The teams getting this right aren't picking a side. They're treating fine-tuning as the layer for behaviour and RAG as the layer for facts, and running both:

  1. RAG holds the truth. Policies, product specs, customer data live in a retrieval layer, kept current with a database write, not a training run. When the doc set is small enough, a cached prefix does the same job.
  2. A small fine-tuned model holds the style. A small open-weight model (or Gemini's tunable Flash tier on Google's Agent Platform) tuned only on voice and formatting, cheap to run, cheap to retrain.
  3. They're wired together, not bolted on: RAG retrieves, passes a compact structured context to the tuned model, and the output comes back with both consistent voice and a traceable source.
The hybrid stack: retrieval holds the facts, a small tuned model holds the style Flow from left to right, then down. A user question goes to the facts layer: RAG retrieval, or a cached prefix if the documents fit in the prompt. The facts layer passes compact, structured context to a small fine-tuned model that holds voice and format. The output is an answer with consistent voice and a cited source. Facts belong to RAG or caching; behaviour belongs to fine-tuning. A review gate sits at the retrieval step and at the generation step. FACTS = RAG / CACHING Question from the user Facts layer RAG retrieval, or a cached prefix if the docs fit Context compact and structured review gate BEHAVIOUR = FINE-TUNING Small fine-tuned model holds voice and format Answer with citation consistent voice, cited source review gate RAG holds the truth. A small fine-tuned model holds the style.
The hybrid stack: retrieval holds the facts, a small tuned model holds the style Flow from top to bottom. A user question goes to the facts layer: RAG retrieval, or a cached prefix if the documents fit in the prompt. The facts layer passes compact, structured context to a small fine-tuned model that holds voice and format. The output is an answer with consistent voice and a cited source. Facts belong to RAG or caching; behaviour belongs to fine-tuning. A review gate sits at the retrieval step and at the generation step. User question Facts layer FACTS RAG, or a cached prefix if docs fit Compact structured context Small fine-tuned model BEHAVIOUR holds voice and format Answer with citation Review gate at retrieval and at generation
Fig. 3 — The hybrid stack: retrieval or a cached prefix holds the facts, a small fine-tuned model holds the style, and each retrieval and generation step is a place for a review gate.

This is also where the human-in-the-loop question gets answered concretely: every retrieval and every generation step is a place to put a review gate, not a place to hope the model got it right. Nothing here has to be a black box, and any developer inheriting the system should be able to read what it does.

What if you already fine-tuned for facts?

Pivot in four steps, without scrapping the work:

  1. Audit. List all fine-tuned models. Classify them: information-heavy or behaviour-heavy?
  2. Restructure. For information-heavy models, move the facts into the prompt (static first, dynamic last) and turn on provider caching, or into a retrieval layer if they don't fit.
  3. Measure. Track cache hit rate, cache invalidation frequency and cost per query. Run your eval suite to check quality matches the old fine-tuned model. If you don't have one, build it before you train anything again.
  4. Scale. Roll out to all traffic and deprecate the expensive, information-heavy fine-tuned models.

Does RAG alone satisfy EU AI Act requirements?

Not automatically. RAG gives you the raw material for compliance: citable sources and a data layer instead of opaque weights. But the EU AI Act's high-risk system obligations require logging, human oversight, and documented conformity, not just a retrievable source, with fines up to €15 million or 3% of global annual turnover under Article 99. The Digital Omnibus on AI, published in the EU's Official Journal on 24 July 2026, moved standalone Annex III high-risk obligations from 2 August 2026 to 2 December 2027 (EUR-Lex). Treat the new date as "plan for it," not "wait and see."

A RAG pipeline with good logging and an approval gate gets you most of the way there. RAG with no governance layer on top gets you a citable hallucination, which is not the same thing as compliance.

How does ProNova build this with clients?

We don't sell "fine-tuning or RAG" as a product. The work takes one of three shapes:

  • Custom AI agents: we build the governed pipeline. Connectors, retrieval, human review gates and a spending limit on every run. The agent runs alongside your team before it acts on its own, then runs in your own account, with one person on your team trained to manage it.
  • Custom MCP connectors: when the official connector can't reach the data your retrieval layer needs, we build the missing tools next to it.
  • Team AI training: live training on your team's real tasks, so your team leaves with a workflow map, shared prompts and the rules for using them.

Measure the baseline before you touch anything. That's not a compliance step, it's how you find out whether fine-tuning, RAG, or neither is actually your cost problem before you spend on either one.

If your AI bill moved and you can't fully explain why, book a scoping call and we'll help you find the driver before you commit to an architecture.

FAQ

Is RAG always cheaper than fine-tuning?

Not always, but usually for workloads where facts change faster than a retraining cycle. RAG shifts cost to ongoing tokens and retrieval infrastructure instead of training runs, which tends to win below very high, stable query volumes.

Can I use both fine-tuning and RAG together?

Yes, and for most enterprise workloads it's the stronger design: a fine-tuned small model for voice and format, fed structured context from a RAG layer for facts, so you get consistent tone and traceable answers together.

Does fine-tuning make a model more auditable?

No. Fine-tuning changes model weights, which are not inspectable, so you can't natively trace a specific output back to a source document. RAG's retrieved-document trail is what supports an audit request; fine-tuning's consistency does not substitute for it.

What actually causes AI cost overruns, if not the model choice?

Governance gaps, most often. DoiT/Sapio's 2026 survey found split accountability between technology and finance teams, and only 15% of finance leaders able to calculate AI ROI without significant bottlenecks, which is a structural problem no architecture choice fixes on its own.

Topics
Author
Izzy Aly · Co-Founder, Systems and Architecture
Share

Keep reading

All posts →