Fine-Tuning vs RAGvs Prompting in 2026

Most arguments about fine-tuning versus RAG versus prompting are arguments about three things that do not compete. They operate at different layers of the system, they fail in different ways, and they are cheap or expensive at different times. Once you separate them by mechanism, the decision mostly makes itself.

• Prompting and context engineering change behavior within a single call: Nothing about the model changes. You are choosing which tokens occupy a finite attention budget, and in what order. This is where instructions, exemplars, tool definitions, output schemas and retrieved passages all live. It is the cheapest layer to change and the only one you can revert in a single commit.
• Retrieval changes what the model knows at inference time: Facts arrive as tokens. Nothing is memorized; the model is simply reading. That is precisely why retrieval handles changing facts, per-user facts and access-controlled facts, none of which a set of weights can represent without being retrained.
• Fine-tuning changes the weights: It reliably buys output format, house style, domain vocabulary, and the ability to run a smaller model at lower latency and unit cost. It buys facts unreliably, and — as the next section shows — it can degrade the model's willingness to say it does not know.

This article is written for a CTO or staff engineer deciding where to spend the next six weeks. It is deliberately honest that prompting plus retrieval solves the large majority of cases, and that the most common expensive mistake in applied LLM work is a fine-tuning project started before anyone measured a retrieval baseline.

Every price, benchmark and model reference below is current as of August 2026 and carries its source. Model pricing in particular changes monthly; re-verify anything you are about to put in a budget.

This is the claim that reframes the whole decision, so it is worth being precise about what the evidence does and does not support.

The general finding first, stated as an engineering conclusion rather than a single measured delta: fine-tuning reliably moves format, style, vocabulary and latency, and unreliably moves facts. The mechanism is intuitive once you see it. A fine-tuning set of question-and-answer pairs teaches a model what a confident, well-formed answer looks like in your domain. It does not, in any dependable way, install the specific facts inside those answers as retrievable knowledge. So the model gets better at producing answer-shaped text without getting proportionally better at recognizing the boundary of what it knows — a trade that is bad in most enterprise settings and catastrophic anywhere a wrong answer costs more than no answer.

We deliberately state that as a general finding. Specific percentage-pair claims about fine-tuning and hallucination circulate widely in this discussion, and several of them do not survive primary-source checking, so we do not repeat them.

What is measured, and what turns out to be far more useful for your decision, is a sharper question: is the retrieved material even enough to answer with? Google's Sufficient Context work (ICLR 2025) asked exactly that, and its findings reframe where the failure actually lives.

• Models answer anyway when the context is not enough: State-of-the-art LLMs output correct responses only 35% to 62% of the time with insufficient context — which means they are producing answers in a regime where the material to answer correctly was never supplied.
• The failure mode is assertion, not silence: The paper's verbatim finding: with RAG, models hallucinate more than abstain. Retrieval does not make a model cautious; it makes a model confident, whether or not the retrieval worked.
• Benchmark corpora are far less well-retrieved than teams assume: On Musique, 55.4% of instances have insufficient context. If a curated research benchmark is at that level, an enterprise corpus assembled under deadline is unlikely to be better.
• Model size changes which failure you get, not whether you get one: Larger models "excel at answering queries when the context is sufficient, but often output incorrect answers instead of abstaining when the context is not." Smaller models "hallucinate or abstain often, even with sufficient context." Scaling up the model relocates the problem rather than removing it.

Read that list against a fine-tuning proposal. If a meaningful share of your queries never had sufficient context in front of the model, then no modification of the weights addresses them — the material simply was not there. That is a retrieval problem wearing the costume of a model problem, and it is the single most common misdiagnosis we are called in to correct.

Fine-tuning changes how a model speaks. Retrieval changes what it can speak about. Confusing the two is the most expensive mistake in applied LLM work, because you find out six weeks and one training budget later.

— Frenchy Digital engineering principle

None of this means fine-tuning is useless. It means fine-tuning has a job, and knowledge injection is not that job. The section on when fine-tuning actually wins lays out the four cases where it earns its cost.

Here is the same argument as a table, which is the form most engineering teams actually use when they are choosing. Read the "iteration speed" row twice — it is the dimension that decides most projects and the one that never appears in vendor comparisons.

Fine-tuning versus retrieval versus prompting, compared by mechanism rather than by marketing — Frenchy Digital, August 2026.

Two observations from that table are worth pulling out. First, the three techniques compose. A production system almost always runs prompting plus retrieval, and a mature one sometimes adds a narrow fine-tune on top for a specific, measured reason. The question is never which one, it is which one next.

Second, the cost columns are not comparable in kind. Prompting and retrieval costs are marginal and continuous — you can turn them up and down weekly. Fine-tuning cost is lumpy and largely sunk: dataset construction dominates, and it has to be redone when the base model changes or when the target behavior drifts. That asymmetry is why the decision procedure later in this article puts fine-tuning at step seven rather than step one.

Worth stating plainly, because it shifts the practical default: OpenAI is winding down its fine-tuning platform and has closed it to new users, as reflected on its API pricing and platform documentation as of August 2026.

That is a market signal, not a proof that fine-tuning is wrong. But it changes what a fine-tuning plan looks like in practice. If one of the two largest managed platforms is no longer accepting new fine-tuning customers, then a team starting today is mostly choosing between a smaller set of managed offerings and fine-tuning open-weight models it hosts itself. Self-hosting is a legitimate path — it is also a serving bill, a GPU capacity problem, an evaluation burden and an upgrade treadmill that a prompt and an index simply do not carry.

Set that against the direction of inference pricing. Epoch AI found that the price of reaching a fixed capability level has fallen between 9× and 900× per year depending on the benchmark, with GPT-4-level GPQA Diamond performance falling roughly 40× per year. That analysis was published in March 2025 and we could not verify a 2026 update, so treat it as directional rather than current. The direction is still the point: the alternative to a fine-tune — buying a slightly better model, or the same capability at a lower tier — keeps getting cheaper, while a fine-tune depreciates from the day it is trained.

Anthropic's context engineering guidance supplies the most useful mental model available for this layer: treat context as an attention budget rather than a container. Because transformers compute relationships across every pair of tokens, the cost of attention grows quadratically, and accuracy degrades as context grows — well before the window is full. The name for that degradation is context rot, and it is the reason "just put everything in the prompt" stops working long before you hit a token limit.

The practices that follow from it are concrete:

• Compaction: Summarize history as it grows, keeping architectural decisions and unresolved problems while discarding the transcript that produced them. Anthropic recommends server-side compaction as the primary strategy rather than an optimization of last resort.
• Just-in-time retrieval over preloading: Fetch what the current step needs instead of front-loading everything the task might need. Hybrid approaches are appropriate where latency is critical and the data is small and stable.
• Structured note-taking: External memory files that persist knowledge across context resets, so a long-running task does not depend on everything staying in the window.
• Sub-agent isolation: Let a sub-agent hold detailed search context privately and return a condensed 1,000 to 2,000 token summary to the lead. The lead's window stays high-signal.
• Tool design discipline: Anthropic's test is the right one: if a human engineer cannot definitively say which tool should be used, an agent cannot be expected to do better. Ambiguous tool surfaces are a context problem wearing a different hat.
• The guiding principle: The smallest set of high-signal tokens that maximizes the outcome. Not the most tokens the window allows.

Now the economics, because they are what actually settle the "fine-tune so we can shorten the prompt" argument. A long, stable prefix is not expensive if it is cached. Anthropic's published pricing as of August 2026 makes the arithmetic explicit.

Prompt caching and context-reduction economics on Anthropic's published pricing, as of August 2026. Re-verify before budgeting — these figures change.

One more cost mechanic that invalidates old baselines: Claude 4.7 and later, including Sonnet 5, use a tokenizer that produces roughly 30% more tokens for the same text.The per-token price did not change; the cost per request did. If you are comparing this quarter's spend against last year's model, re-baseline with a token-counting call rather than applying a multiplier.

If prompting is the first lever, retrieval is the second, and the measured gains available here are larger than most teams realize because most teams stop at "we embedded the documents." What follows is the decision-relevant summary; the full treatment of indexing, access control, vector-store selection and staleness lives in our guide to RAG for enterprise knowledge bases.

Start with the finding that surprises people. On an independent 2026 benchmark of 23,088 queries over 7,318 text-and-table financial documents, BM25 keyword search beat dense embedding retrieval outright — Recall@5 of 0.644 versus 0.587 for text-embedding-3-large. Hybrid fusion of the two reached 0.695. That is a useful corrective to the reflex that embeddings supersede keyword search; on corpora full of identifiers, codes and exact numbers, they frequently do not.

Retrieval interventions ranked by measured effect, with the harness behind each number — compiled August 2026.

Two caveats belong with that number, because the article is worth less without them. The benchmark covers one narrow domain — 145,000 Amazon reviews, 300 queries — it reports Hit@1 rather than NDCG, and it tested no commercial reranking APIs. What makes it credible is that it points the same direction as everything else: the independent financial-document benchmark also found reranking the largest single jump, from 0.695 to 0.816 Recall@5.

On fusion, one correction to a widely repeated claim. Reciprocal rank fusion is a sensible default and is what Elasticsearch, OpenSearch, Qdrant, Weaviate and Milvus ship; it delivers a real 5 to 8 point gain. Independent work has shown that convex combination can beat RRF in and out of domain and that RRF is sensitive to its parameters — but that is an argument for tuning fusion, not for abandoning it.

Then there is the technique with the best cost-to-benefit ratio in the whole pipeline. Contextual Retrieval prepends a short generated context — 50 to 100 tokens — to each chunk before embedding, so the chunk carries the document context it would otherwise lose. In Anthropic's published measurement, top-20 retrieval failure fell from a 5.7% baseline to 3.7% with contextual embeddings, to 2.9% adding contextual BM25, and to 1.9% with reranking on top. The one-time preprocessing cost is $1.02 per million document tokens. A ten-million-token corpus costs about ten dollars to preprocess. Two notes: it was published in September 2024, so it is not a new technique, and the measurement is the vendor's own.

Hybrid BM25 plus dense, fused with RRF. Rerank with a small cross-encoder. Two-hundred-token chunks with contextual prefixes. Skip HyDE. Skip semantic chunking. Treat graph-based retrieval as a multi-hop-only escalation, not a default.

— The stack the evidence actually supports

The most valuable part of the 2026 retrieval literature is not what works. It is the list of popular techniques that measure at or below the baseline they were meant to improve. Every one of these is currently being sold somewhere as advanced RAG.

Retrieval techniques whose measured results do not justify their cost — compiled from independent benchmarks, August 2026.

HyDE deserves particular attention because it is so widely recommended. The idea is elegant: have the model hallucinate a plausible answer document, embed that, and retrieve against it. On the 2026 financial-document benchmark it scored Recall@5 of 0.544 — worse than simply embedding the query, at 0.587. The original paper claims it significantly outperforms Contriever but publishes no numeric deltas of its own, which is why the claim survived so long without being checked.

Multi-query expansion tells a similar story: 0.640 Recall@5, statistically indistinguishable from plain BM25 at 0.644, for several times the token cost and an extra model call per query. Expansion also backfires on precise numerical queries — which describes most of what an enterprise corpus is actually asked.

More retrieval depth is not more knowledge, and the cleanest evidence isolates exactly why. In the Lost in the Middle experiments, increasing the number of retrieved documents from 20 to 50 lifted retriever recall substantially — from roughly 69 to 88 — while reader accuracy barely moved: about 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3. The retriever was finding more of the right material and the model was not using it.

That is the whole argument against paying for depth, stated as a mechanism rather than an anecdote. Reader performance saturates long before retriever recall does, so every document past the saturation point is tokens you buy and attention you spend for nothing. Combine it with RULER's finding that models claiming 32K or more context frequently cannot maintain performance at 32K, and the design conclusion is unambiguous: retrieve enough that the answer is somewhere in the candidate set, rerank hard, and pass few. That is also precisely why reranking outperforms deeper retrieval — it improves what reaches the model rather than how much does.

Having spent five sections arguing against the default reach for fine-tuning, here is the honest other side. There are four cases where it earns its cost, and they share a property: the target is a behavior, not a fact.

• 1. Format and structural adherence at volume: You need output in a fixed schema, millions of times, and a schema in the prompt plus structured-output enforcement is measurably still failing. Fine-tuning is very good at this, and the win compounds because the prompt gets shorter — though check the caching math first, since a cached prefix reads at 0.1× base input and may already have solved the cost half of the problem.
• 2. Style, tone and voice that resist instruction: House style is notoriously hard to specify in prose and easy to demonstrate in examples. If you have thousands of examples of the voice you want and instructions keep drifting, weights are the right place to put it.
• 3. Domain vocabulary and jargon: Specialist terminology the base model tokenizes awkwardly or handles with visible unfamiliarity — legal drafting conventions, clinical shorthand, industrial part taxonomies. This is closer to knowledge than the other three, but it is knowledge about language rather than about facts, and it is stable over time.
• 4. Latency and unit cost via a smaller model: Distilling a specific behavior from a large model into a small one you serve yourself. This is the strongest quantitative case, because the saving is measurable in advance: compare the monthly inference bill on the large model against the small model plus serving infrastructure plus the amortized cost of retraining on the next base model. It is also the case that overlaps most with on-device deployment, where a small tuned model may be the only option available.

What none of those four cases is: making the model know your documents, keeping the model current with facts that change, or fixing hallucination. If the proposal in front of you is one of those three, the evidence in this article says redirect it.

Abstract comparisons do not change decisions; a spreadsheet does. Here is a concrete workload costed against published pricing as of August 2026, so a fine-tuning proposal can be compared against simply buying a better model or adding a reranker. If the wider question is whether to build any of this in-house at all, our build versus buy analysis covers the other half of the decision.

The workload: an internal assistant handling 200,000 requests per month. Each request sends 6,000 input tokens — a 4,500-token stable prefix of system prompt, tool definitions and exemplars, plus 1,500 tokens of retrieved context — and produces 400 output tokens. That is 1.2 billion input tokens and 80 million output tokens per month.

Cost model for a 200,000-request-per-month workload against published pricing as of August 2026. Sonnet 5 introductory pricing runs through August 31, 2026, after which it moves to $3 input and $15 output per million tokens — which would put Path A at $4,800 per month.

Read the table as a ranking of leverage. Caching alone halves the bill for about a day of work and changes nothing about output quality. Reranking costs $200 a month and, in the benchmark that measured it, moved Hit@1 twenty points. Contextual prefixes on a ten-million-token corpus cost about ten dollars, once, and cut top-20 retrieval failure by two-thirds when combined with reranking. Model tier is a 2.5× lever between Sonnet 5 and Opus 5 and better than an order of magnitude down to a small model — but only your eval set can tell you whether the small model holds.

Two pricing mechanics that belong in any model of this kind. Anthropic applies no long-context premium — a 900,000-token request bills at the same per-token rate as a 9,000-token one — while Google's Gemini Pro models double input price above 200,000 tokens, so a long-context architecture has materially different economics depending on provider. And reasoning tokens bill as outputtokens on both major providers, the expensive side of the ledger. On Claude Opus 5, omitting the thinking parameter now runs adaptive thinking where the same request on Opus 4.8 ran with none; migrated code can silently acquire thinking spend. On GPT-5.6, the reasoning context defaults to all turns rather than the current turn, so a multi-turn agent's reasoning cost compounds quietly. Both are worth an hour of verification before you sign off a forecast.

Everything above is a hypothesis about your system. The decision procedure in the next section only works if you can tell whether a change helped, and that requires an eval set built before you start changing things.

The most important measurement finding for this decision is uncomfortable: retrieval metrics do not predict answer quality. The eRAG work found only small correlation between traditional retrieval metrics and downstream generation quality, improving Kendall's tau by 0.168 to 0.494 when retrieval was scored against what the generator actually did with it. UDCGstates it more bluntly — nDCG, MAP and MRR "do not accurately predict RAG performance" — and improves correlation by up to 36% with a utility-based measure.

The reasons are mechanical, and they are the same reasons that make reranking valuable. A language model processes retrieved documents as a whole rather than sequentially, so the positional discounting baked into nDCG and MRR is measuring something the consumer of the ranking does not experience. And related-but-irrelevant documents actively degrade output — a distractor at rank 3 can be worse than an empty slot. A pipeline optimized purely for nDCG can therefore get measurably worse at answering questions.

Automated RAG evaluation frameworks help but are not oracles. RAGAS remains actively maintained, with v0.4.3 released in January 2026. Under a metamorphic testing regime with more than 28,000 corpus mutants, however, its best metric reached an F1 of 0.570 at fault detection against a metamorphic oracle's 0.927 to 1.000. Use these tools to triage, and use human-labeled outcomes to decide.

The discipline itself is well documented. Anthropic's guidance and LangChain's agree on the shape of it:

Evaluation discipline for choosing between prompting, retrieval and fine-tuning — synthesized from published guidance, August 2026.

Add one diagnostic that is specific to this decision and takes an afternoon to build: the oracle-context test. Take your failing cases and manually place the correct source material in the context window. If the model then answers correctly, your problem is retrieval and no amount of fine-tuning will fix it. If it still answers incorrectly with perfect context, your problem is generation — and that is the narrow region where fine-tuning is a candidate. That single split does more to route engineering effort than any leaderboard.

Finally, score cost alongside accuracy. Princeton's Holistic Agent Leaderboard — 21,730 rollouts across nine models and nine benchmarks — found that higher reasoning effort reduced accuracy in the majority of runs, and its live board notes that agents can be 100× more expensive while only 1% better. An accuracy-only eval will happily recommend the configuration that bankrupts the feature.

This is the procedure we run on client engagements, rendered as a table because that is the form people actually follow. Work it top to bottom. Do not skip step 4.

The Frenchy Digital decision procedure for prompting versus retrieval versus fine-tuning, 2026.

Steps 1 through 3 are a diagnosis, and most teams have never made one. "The output is bad" is not a diagnosis; "the model does not have the fact," "the model has the fact but formats it wrong," and "the model loses the instruction once the context exceeds twenty thousand tokens" are three completely different problems with three completely different fixes. Step 5 — the oracle-context test — is the one that separates a retrieval problem from a generation problem, and it is the cheapest high-value experiment in this entire article.

Steps 6 and 7 exist because a fine-tune without an eval set is unfalsifiable and a fine-tune without a migration plan is a liability with a launch date. If you can answer both, fine-tuning is a reasonable engineering decision. If you cannot, it is a purchase.

Every one of these has appeared in a real vendor conversation or a real architecture review. The first is the one that matters most.

The Frenchy Digital red-flag list for model-strategy proposals, 2026.

A proposal to fine-tune, arriving before anyone has measured a hybrid-plus-reranker retrieval baseline, is a proposal to spend the most money on the least reversible option to fix a failure nobody has diagnosed. Ask for the baseline. If the answer is that they have not measured one, you have your answer about the proposal too.

— Frenchy Digital buyer's principle

These are the bands Frenchy Digital uses to scope AI engineering work in 2026. The first one exists specifically because the decision in this article is worth making before the build, not during it.

Frenchy Digital cost bands for AI agent and retrieval engagements, 2026.

Senior-led delivery runs $150 to $225 per hour and ongoing retainers run $2,500 to $9,500 per month, covering model and dependency upgrades, eval expansion, incident response and a quarterly technical review. Every engagement carries a 30-day post-launch warranty, and you receive a written scope with a fixed-price phased proposal within 5 business days of the discovery call.

One budgeting note. The discovery engagement is the cheapest band and it is the one that most often changes the plan — usually by replacing a proposed fine-tuning project with two weeks of retrieval work and a caching pass, at a fraction of the cost and with a measurement attached. That is not a sales position; it is what the evidence in this article keeps producing.

The argument above is as strong as the evidence behind it, and the evidence has edges. Here is where it is thin, contested, or perishable.

• The reranker benchmark is one narrow domain: The 62.67% to 83.00% Hit@1 result comes from 145,000 Amazon reviews and 300 queries, reports Hit@1 rather than NDCG, and tested no commercial reranking APIs. It points the same direction as the independent financial-document benchmark, which is why we cite it — but it is not a general law, and your corpus may rank rerankers differently.
• The hybrid retrieval numbers are one corpus: 23,088 queries is a substantial sample, but they are all text-and-table financial documents. BM25 beating dense retrieval is plausible partly because that corpus is dense with identifiers and exact figures. On narrative prose the ordering may reverse. Measure on yours.
• Contextual Retrieval is a vendor's own measurement, from 2024: The 5.7% to 1.9% figures and the $1.02 per million document tokens are Anthropic's, published in September 2024 using Claude 3 Haiku. It is not a 2026 technique and it has not been independently replicated at that scale to our knowledge. The mechanism is sound and the cost is trivial, which is why we still recommend testing it.
• The knowledge-injection claim is a general finding, not a single measured delta: We state that fine-tuning is a poor mechanism for injecting knowledge as an engineering conclusion, supported by the decision logic in this article and by the broader literature — not as one benchmark number. Specific percentage-pair claims about fine-tuning and hallucination circulate widely and several do not survive primary-source checking, so we do not repeat them. If you need a number for your own decision, measure sufficiency on your own corpus and compare against your own baseline.
• The saturation figures come from an earlier model generation: The 20-to-50-document result is from Liu et al.'s Lost in the Middle work and was measured on GPT-3.5-Turbo and Claude-1.3. The mechanism it isolates — reader performance saturating long before retriever recall does — is corroborated by every reranking result in this article, which is why we rely on it. Whether the exact saturation point has moved on 2026 models is contested rather than established, and we could not verify a replication.
• Chunking has no universal winner: One 2026 study found structure-aware chunking with title-chain prefixes lifting MRR@5 from 0.374 to 0.463 over 1,600 production queries; another found fixed-size chunking beating structure-aware on a different domain. Anyone telling you there is a correct chunking strategy has not tested two domains.
• Prompt injection is not solved, and retrieval widens the surface: Every retrieved document is untrusted input that the model may read as instruction. No mitigation described here — or anywhere — makes an agent safe. The correct framing is defense in depth and blast-radius reduction: allowlisted tools, deny-by-default arguments, scoped retrieval, and injection cases in CI. Fine-tuning does not help with this either.
• Prices are the most perishable content on this page: Every figure is as of August 2026 and several are explicitly time-boxed — Sonnet 5's introductory rate ends August 31, 2026. The tokenizer change in Claude 4.7 and later means a 30% token increase for the same text, so historical cost baselines do not transfer. Re-verify before budgeting.
• Per-task cost data is thin and mostly vendor-published: Credible independent token-per-task figures for production agents barely exist. The multipliers in circulation — agents using roughly 4× the tokens of chat, multi-agent systems roughly 15× — come from a vendor's own engineering write-up. Treat them as directional and measure your own.

None of that argues for paralysis. It argues for the same thing the rest of the article argues for: run the cheap experiment, measure it against a set of real failures, and let the result rather than the roadmap decide what you build next.