RAG for EnterpriseKnowledge Bases in 2026

Someone on your leadership team has read that retrieval-augmented generation is finished, that context windows got long enough to make it unnecessary, and that you should paste the documents into the prompt and move on. Someone else has read that agentic RAG is the new thing and that your pipeline is obsolete. Both of these are partially true, and neither is a plan.

Here is the honest position, and it is the one the published evidence supports. No credible source says RAG is dead. What is obsolete is naive RAG — the 2023 pattern of embedding everything, retrieving the top five chunks by cosine similarity, stuffing them into a prompt, generating once, and never checking whether the retrieval was any good. That pattern underperforms, it fails silently, and it degrades as your corpus grows.

Retrieval itself survives, and it survives in a specific shape: as a tool inside an agent loop. The model decides whether to search, chooses which index to search, judges whether what came back is sufficient, re-queries when it is not, and refuses when it still cannot answer. Classic RAG is a pipeline with no agency over whether or what to retrieve. Agentic RAG is a control loop with a policy.

This article is written for an engineering lead who has been handed the words "build a knowledge base with AI" and needs to decide what to actually build. It is deliberately evidence-led, and the most useful sections are the ones reporting negative results — the techniques that appear in every architecture diagram and do not survive contact with a benchmark. Every number below carries its source and its date. Where a widely circulated figure could not be traced to a primary source, we say so and leave it out rather than launder it.

The question is never "RAG or long context." It is: what is the smallest amount of the right text you can put in front of the model, how do you know it is the right text, and what happens when it is not?

— Frenchy Digital retrieval principle

Start with the benchmark that measures the claim directly. RULERevaluated 17 models across 13 tasks and reached a blunt conclusion: the models "all claim context sizes of 32K or greater, [but] only half can maintain satisfactory performance at 32K." They are near-perfect on vanilla needle-in-a-haystack retrieval and drop substantially as length and task complexity rise together. A advertised context window is a capacity, not a capability.

The mechanism was described earlier, in Lost in the Middle (Liu et al., TACL). Multi-document question answering degrades depending on where the relevant passage sits in the context: performance "can drop by more than 20%," and in the worst case, performance in the 20- and 30-document settings is lower than performance without any input documents at all — that is, worse than closed-book, at 56.1% for GPT-3.5-Turbo. Against an oracle setting that supplies only the single relevant document, the same model reaches 88.3%.

Multi-document QA bounds from Lost in the Middle, Table 1. The gap between these two columns is the entire value of retrieval; the failure is what happens between them.

The finding that should change your architecture is the pairing in that table. An extended-context version of a model does not use context better than its shorter sibling.GPT-3.5-Turbo and GPT-3.5-Turbo-16K are described as "nearly superimposed." Claude-1.3 scores 48.3 closed-book and 76.1 oracle; Claude-1.3-100K scores 48.2 and 76.4. Roughly six times the window bought nothing measurable on this task. Buying a longer window is not the same as buying better use of it.

Three mechanism findings from the same paper are worth carrying into design reviews. The U-shaped position curve is not caused by instruction tuning — base MPT-30B shows it too, and instruction tuning only narrows the best-versus-worst spread from roughly 10% to roughly 4%. It is scale-dependent: Llama-2 7B is solely recency-biased, and the U-shape appears only at 13B and 70B. And it emerges beyond training length: Flan-UL2 evaluated inside its own 2,048-token training window shows only a 1.9% absolute spread. Positional weakness is a property of extrapolation, not of transformers as such.

The paper also tests the obvious workaround. Query-aware contextualization— placing the question both before and after the documents — makes synthetic key-value retrieval near-perfect but "minimally affects" multi-document QA. The prompt-engineering fix does not fix the real task.

So: long context wins on bounded, static, single-document tasks. One contract, one filing, one specification, read end to end, where the whole document is relevant and fits comfortably. The moment the input is a corpus rather than a document — many sources, mixed relevance, changing daily, governed by permissions — you are doing retrieval whether or not you call it that. The only question is whether you are doing it deliberately.

This is the centerpiece of the article. The following numbers come from an independent 2026 evaluation over 23,088 queries against 7,318 text-and-table financial documents drawn from FinQA, ConvFinQA and TAT-DQA. It is the largest cleanly reported comparison of retrieval configurations we could verify, and it produces one result that contradicts how most enterprise RAG systems are built.

Retrieval configurations compared across 23,088 queries over 7,318 text-and-table financial documents (arXiv 2604.01733, April 2026). Independent evaluation.

Read the first two rows again. BM25 — a lexical scoring function from the 1990s — beat dense retrieval with text-embedding-3-large on this corpus, 0.644 against 0.587 on Recall@5, and on every other reported metric. That is the corrective to "just embed everything." Semantic similarity is genuinely useful for paraphrase, synonymy and conceptual queries. It is worse than keyword matching when the query contains an exact identifier, a product code, a defined term, a table header, or a number — which describes a great deal of enterprise search.

Fusing the two channels with reciprocal rank fusion reached 0.695 — a real gain of roughly five to eight percentage points, with the largest single improvement of +8.1 points on Recall@5 observed on the table-heavy TAT-DQA subset. RRF is generally available in Elasticsearch (with a rank_constant of 60), OpenSearch, Qdrant, Weaviate and Milvus, which means for most teams this is a configuration change rather than a project.

One methodological note in the interest of accuracy: independent work on fusion functions finds that a convex combination of normalized scores can beat RRF both in-domain and out-of-domain, and that RRF is sensitive to its parameters. That paper says RRF is beatable, not that RRF fails. RRF remains the sensible default because it needs no score normalization and no tuning; if you have the evaluation harness to tune a weighted combination, tune it.

The honest limitation on this table: it is one corpus, and it is financial text with a heavy table component. Lexical matching is unusually strong on numeric and tabular content. On a corpus of long narrative prose with high vocabulary variation, dense retrieval will close much of that gap. What transfers is not the exact ordering but the structural conclusion — you cannot know which channel wins on your corpus without measuring, and running both costs less than being wrong about one.

If you do one thing to an existing RAG system this quarter, do this one. Retrieve a wide candidate set — top 100 is typical — then score every candidate against the query with a cross-encoder that reads the query and the passage together, and keep the top ten. Bi-encoder retrieval is fast because it never compares query and document directly; a cross-encoder does, which is why it is more accurate and why you can only afford it on a shortlist.

An independent evaluation published in February 2026 measured this on 145,000 Amazon reviews with 300 queries, using e5-base to retrieve the top 100 and reranking down to the top 10, on an H100. The baseline Hit@1 was 62.67%.

Cross-encoder reranker comparison, February 2026 — 145k Amazon reviews, 300 queries, e5-base top-100 to top-10, H100. Independent evaluation.

Two things stand out. The first is the size of the win: +20.33 percentage points on Hit@1 from a 149-million-parameter model. Nothing else in this article delivers that for that little. The second is the shape of the curve: bigger is not better. The 4-billion-parameter qwen3-reranker-4b scored 77.67% — lower than the 149M model — while taking over 1,000 milliseconds per query against roughly 150 to 170. A billion-parameter model matched the small one exactly at 83.00% but took 243ms. On this evidence, the cost-quality frontier for reranking sits at the small end.

Take the caveats seriously before you generalize: one narrow domain (product reviews), Hit@1 rather than NDCG, and no commercial reranking APIs were tested. Budget roughly 150 to 250 milliseconds per query for a self-hosted reranker and decide whether your latency envelope absorbs it. For most internal knowledge-base workloads it does, comfortably.

Pricing, as of August 2026, with an accuracy warning. Voyage publishes rerank-2.5 at $0.05 per million tokens and rerank-2.5-lite at $0.02 per million. Pinecone publishes rerank at $2 per 1,000 requests. ZeroEntropy publishes zerank-2 at $0.025 per million tokens; Mixedbread publishes $3.50 per 1,000 queries. Cohere no longer publishes per-search API pricing at all — only hourly Model Vault rates (Rerank 4 Fast at $5.00/hr; Rerank 4 Pro at $5.00/hr Medium and $10.00/hr Large). Rerank 4 was announced December 11, 2025 with a 32k context window, up from 4,096 in v3.5.

The reason to trust reranking more than anything else in this article is convergence. The 2026 evaluation above says add a reranker. The 2023 Lost in the Middle authors, working on an unrelated problem with unrelated models, independently recommend reranking as the remedy for positional degradation. And the hybrid benchmark in the previous section moved from 0.695 to 0.816 on Recall@5 by adding one. Three separate lines of evidence, one conclusion.

This is the most valuable section in the article, because it is the part nobody writes. Every RAG architecture diagram on the internet includes query rewriting and semantic chunking. Both have measured evidence against them, from independent sources, and removing them makes your system faster, cheaper and — on the available data — no worse.

Retrieval techniques with negative or null measured results. Hybrid and rewriting figures from the same 23,088-query benchmark; chunking figures from a NAACL Findings 2025 study and Chroma's public evaluation.

HyDE is the headline. The technique — generate a hypothetical answer document, embed that, and search with it — is elegant, widely implemented, and on the same benchmark that produced the hybrid table above it scored Recall@5 of 0.544, below plain dense retrieval at 0.587. It is not a marginal gain that fails to justify its latency. It is worse than doing nothing, on that corpus, while adding a full generation round trip to every query. The original HyDE paper claims it significantly outperforms Contriever but publishes no numeric deltas, which is worth knowing before you build on it.

Multi-query expansion — generate several phrasings, retrieve for each, merge — scored 0.640, effectively identical to BM25 alone at 0.644, for several times the cost. The failure mode named in that work is instructive: expansion backfires on precise numerical queries. Broadening a query is helpful when the user was vague and harmful when the user was exact, and enterprise users are frequently exact.

Semantic chunking is the other one to cut. A NAACL Findings 2025 study concludes directly that its "computational costs are not justified by consistent performance gains." Chroma's public evaluation shows why the intuition misleads: LLM-based semantic chunking reached 91.9% recall but only 3.9% precision, against 88.1% recall and 7.0% precision for plain recursive chunking at 200 tokens. Semantic chunkers produce larger, more coherent-looking chunks that contain more irrelevant text — you retrieve slightly more of the right thing wrapped in a lot more of the wrong thing, and the generator pays for all of it.

Recursive chunking at roughly 200 tokens was the most efficient configuration tested. That is a boring answer to a question teams spend weeks on. Take the boring answer, spend the saved time on reranking, and revisit chunk size only if your evaluation set says something specific about your documents.

The one technique in this family worth keeping on the table is late chunking — embedding the full document first and pooling token embeddings into chunks afterward, so each chunk carries document-level context. Published nDCG@10 gains over naive chunking are +6.52 points on NFCorpus, +1.9 on SciFact, +1.34 on TRECCOVID, +0.59 on FiQA, and exactly 0.00 on Quora. Real, modest, and scaling with document length. Useful for long technical documents; not a reason to rebuild your pipeline.

A technique with a plausible mechanism and no published deltas is a hypothesis. Ship the ones with numbers, measure the rest on your own corpus, and delete whatever does not move your evaluation set. Most RAG stacks are carrying two or three components that have never been ablated.

— Frenchy Digital engineering principle

The single best-evidenced chunking improvement is also one of the simplest. A chunk that reads "revenue rose 3% over the prior quarter" is nearly useless in an index — which company, which quarter, which segment? Contextual Retrieval fixes this by generating a short, chunk-specific summary of the surrounding document and prepending it to the chunk before embedding and before indexing it for BM25.

Anthropic published the measurements, and they are staged so you can see what each component contributes.

Contextual Retrieval, published September 19, 2024. Top-20 retrieval failure rate — the share of queries where the correct passage was not in the top 20 results.

The economics are the reason to do it. Preprocessing costs a one-time $1.02 per million document tokens using Claude 3 Haiku to generate a 50-to-100-token prefix per chunk. For a ten-million-token corpus, that is about ten dollars, paid once at ingestion, for a 35% reduction in retrieval failures before you add anything else. Very little in this field has that cost-benefit shape.

Two implementation notes. First, the contextual prefix must go into both indexes — the embedding and the BM25 term index. The table shows contextual embeddings alone taking failure from 5.7% to 3.7%, and adding contextual BM25 taking it to 2.9%; roughly a third of the total gain comes from the lexical side, which teams routinely skip. Second, the final row is reranking again, taking 2.9% to 1.9% for a cumulative 67% reduction. Same intervention, third appearance.

We are deliberately not quoting the prompt-caching arithmetic that usually accompanies this technique. The cache lifetime and the write-versus-read price split are the parts we could not verify to primary sources, and caching behavior changes more often than retrieval quality does. Price the preprocessing at the published per-token figure, treat any caching discount as upside, and re-check the provider's current cache terms before you build a budget on them.

GraphRAG — extracting entities and relationships into a knowledge graph, then answering over the graph — is the most-requested and least-justified item on most enterprise roadmaps. The strongest available signal about its cost comes from Microsoft's own follow-up work.

LazyGraphRAG, published November 25, 2024, reports indexing cost "identical to vector RAG and 0.1% of full GraphRAG," with comparable global-query quality at "more than 700 times lower query cost." That is a vendor self-evaluation and its evidence base is thin — 5,590 news articles, 100 synthetic queries, judged by an LLM. But when the team that published the original method publishes a follow-up saying the original's indexing cost is three orders of magnitude higher than necessary, that is worth more than a competitor saying it.

Independent work in April 2026 gives the balanced read: agentic multi-round dense retrieval narrows the gap to GraphRAG, while GraphRAG retains an edge on complex multi-hop questions and is more stable once the indexing cost is amortized. So the decision rule is a question about your queries, not about your data. If users ask "what changed in the returns policy" — single-hop, lookup-shaped — a graph buys you nothing. If they ask "which of our suppliers are exposed to the same upstream component as the one that failed last quarter" — genuinely multi-hop, requiring traversal across documents that never mention each other — that is where a graph earns its indexing bill.

The genuinely newer technique that is production-ready is learned sparse retrieval. SPLADE-v3, Elastic ELSER v2 and OpenSearch neural sparse have all shipped. These models produce sparse term-weighted vectors — inverted-index-compatible, so they inherit the operational maturity of lexical search — while learning term expansion rather than relying on exact matching. Billion-scale web evaluation published in 2025 found SPLADE beats BM25 on complex queries at higher compute cost, with an expanded-SPLADE variant plus pruning offering the best balance. If you want one thing to pilot beyond hybrid-plus-rerank, pilot this.

Late interaction — ColBERT, ColPali, ColQwen — remains niche relative to hybrid-plus-reranking. Qdrant ships multi-vector support, so trying it is cheap, but no independent 2026 head-to-head with published deltas against a hybrid-plus-rerank baseline exists that we could verify. Treat it as a research direction with real promise for visual and document-image retrieval, not as a decision you can justify with numbers today.

Vector database selection consumes far more meeting time than it deserves. Published pricing as of August 2026, then the recommendation.

Vector database pricing from vendor pricing pages, retrieved August 2026. Verify before committing — pricing pages in this category change frequently.

The default recommendation for a mid-size enterprise: use pgvector if you are already on Postgres. It holds comfortably to roughly 50 to 100 million vectors, keeps your documents, metadata, permissions and embeddings in one data model with one transaction boundary and one backup story, and does not add an operational surface with its own upgrade cadence, its own access model and its own on-call rotation. The joins alone — filtering vectors against permissions and business metadata in the same query — remove an entire category of application-layer bug.

Move off that default for specific reasons. Choose Qdrant or Weaviate when hybrid search with heavy metadata filtering is the dominant access pattern and you want first-class support for it rather than something you assembled. Choose Pinecone or Turbopuffer when you are explicitly buying zero operations, or when you need a large number of isolated tenants and want the vendor to own that problem. Reserve Milvus or Vespa for genuine billion-vector scale, where their complexity is proportionate to the problem.

Embedding model pricing as of August 2026, for the budget: OpenAI lists text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13 per million (3,072 dimensions, 8,192-token context), with the legacy ada-002 at $0.10. Notably, OpenAI has shipped no newer embedding model since the text-embedding-3 line. Voyage lists voyage-4-large at $0.12 per million, voyage-4 at $0.06, voyage-4-lite at $0.02, voyage-context-4 at $0.12 and voyage-code-3 at $0.18. Cohere is on embed-v4.0 with a 128k context window and Matryoshka dimensions of 256, 512, 1024 and 1536, but publishes no per-token API price — secondary sources conflict by an order of magnitude, so we quote none.

On leaderboards: treat MTEB standings as directional at best. Leaderboard rank is measured on public benchmark corpora, not on yours, and the correlation between the two is weaker than the tables imply. The practical move is to take your three shortlisted embedding models, run them against a few hundred of your own labelled queries, and pick on that. It takes an afternoon and it is the only evidence that describes your actual corpus.

Here is where enterprise RAG projects actually die, and it is almost never discussed in the technique literature. Your documents have access controls. The salary bands are visible to HR, the board deck to the board, the incident postmortem to engineering. A retrieval index flattens all of that into one searchable surface, and if the retrieval layer does not reproduce the source system's permission model exactly, you have built a very efficient tool for exfiltrating documents from your own company.

Essentially no vector database enforces end-user document ACLs natively. Every one of them pushes it to the application layer. The one exception we can point at is Databricks AI Search, where the index itself is a Unity Catalog securable and the platform enforces governance — which helps only if your corpus and your governance model already live in Unity Catalog.

Permission enforcement approaches across managed retrieval platforms, as documented in August 2026.

There are exactly two workable patterns, and they trade correctness against latency. Copy ACLs into the index and pre-filter is what Azure AI Searchdoes — fast, self-contained, and eventually consistent with reality. Microsoft's own documentation, updated August 5, 2026, warns of "a timing lag… before changes are recognized," and notes that SharePoint ACL changes inherited from parent scopes require an explicit resync or document reset to take effect. That lag is your exposure window after somebody is removed from a group.

Late binding is the other pattern: check permissions against the source system in real time, at query time. AWS Bedrock Managed Knowledge Bases moved to this model at general availability on July 16, 2026. It costs latency and it puts the source system on your read path — but it is correct at the moment of the query, which is the only moment that matters in an incident review.

The consequences are not theoretical. EchoLeak (CVE-2025-32711, CVSS 9.3), disclosed by Aim Labs, was a zero-click exfiltration path out of Microsoft 365 Copilot. SearchLeak (CVE-2026-42824, disclosed by Varonis on June 15, 2026) scored CVSS 6.5 from Microsoft and 7.5 from NVD. CVE-2026-21520, at CVSS 7.5, affected Copilot Studio and Agentforce. All were vendor-patched. All were in retrieval-connected assistants built by companies with substantial security organizations.

One accuracy note while we are here: the oversharing statistics that circulate alongside these incidents — a widely quoted "40% of organizations delayed rollout" and a specific count of files at risk — appear only in vendor marketing material. We could not trace either to a primary source, so we are not repeating them. The CVEs are enough of an argument on their own.

Almost every team we review treats the vector index as derived telemetry: a pile of floating-point numbers, not a copy of the documents. That assumption is wrong, it is measurably wrong, and it is the most consequential misunderstanding in enterprise RAG.

Vec2Text demonstrated that text can be reconstructed from its embedding by iterative correction: roughly 92% exact recovery of 32-token inputs, with BLEU around 97.3%, operating against text-embedding-ada-002. Later work generalized inversion to arbitrary embedding models zero-shot, without needing to train an inverter per model. An embedding is not a hash. It is a lossy but often reversible encoding of the text.

The 2026 result is worse, and it is about deletion. Ghost Vectors, published June 16, 2026, showed that soft-deleted embeddings remain recoverable from raw HNSW index files, bypassing the database API entirely. On a synthetic health corpus, recovery reached 25.5% for exact names, 46.4% for locations, and 100% for age and gender attributes, with 99% facial identity recovery in the vision case. Most vector databases soft-delete by marking a graph node as deleted and reclaiming the space during a later compaction — an availability decision that was never intended as an erasure guarantee, and is not one.

What to do about it, concretely:

• Classify the index as personal data: Put it in the record of processing activities with a lawful basis, a retention period, and a named owner. If you would not export the source documents to an unlisted third-party service, do not export their embeddings either.
• Make erasure a compaction or rebuild, not a flag: Deletion must reach the on-disk index. Schedule compaction as part of the erasure workflow, or rebuild the affected shard, and record when it completed.
• Verify erasure, do not assume it: After deletion, run the retrieval query that previously returned the record and assert it returns nothing. Store that assertion as evidence — a screenshot of an admin panel is not evidence.
• Treat index files as sensitive artifacts: Snapshots, backups and replicas of an HNSW index are copies of your corpus. They need the same encryption, access control, retention schedule and destruction path as the documents themselves.
• Scope the blast radius of a leaked index: Ask the question directly: if this index file were copied out today, what could be reconstructed from it? The published answer, for short text, is a great deal of it.
• Keep the raw text authoritative: Never let the index become the only copy of anything. Erasure, audit and re-embedding all depend on being able to rebuild from a governed source of truth.

This is not a reason to avoid retrieval. It is a reason to give the vector index the same governance you already give the database it was built from — which most organizations have simply never done, because nobody told them the numbers were text.

Two operational problems that arrive later than you expect and cost more than you budgeted.

Multi-tenancy, where the two market leaders directly contradict each other. Pinecone recommends a namespace per tenant, citing a read-cost difference of 1 read unit versus 100 read units — a hundredfold gap — for a metadata-filtered query against a shared namespace, with the caveats that going beyond roughly 100,000 namespaces requires talking to support and that $in filters are capped at 10,000 values. Qdrant recommends the opposite: a single collection with an indexed tenant payload field, reserving collection-per-tenant for deployments under roughly 500 tenants.

Both are right about their own engine. The cost models are engine-specific and neither argument transfers. The practical consequence is that your tenancy design is not portable: if you architect namespace-per-tenant on Pinecone and later migrate to Qdrant, you are not moving data, you are redesigning isolation. Decide your engine and your tenancy model together, and write down which vendor's cost model your design is optimizing against so the next engineer knows why.

The re-embedding trap. Embedding spaces are not cross-compatible. Vectors from one model are meaningless to another; there is no migration, no conversion, no incremental cutover. A model upgrade forces a full reindex of the entire corpus — and because the old and new indexes cannot be queried together, you need both online during the transition, plus a way to route queries to the right one.

Two design choices reduce this pain considerably. Keep the raw text and its chunking metadata in a governed store you control, so a reindex is a batch job rather than an archaeology project. And prefer models supporting Matryoshka-style truncatable dimensions where available — being able to shorten vectors without re-embedding gives you a storage and cost lever that does not require touching the corpus.

Your retrieval dashboard shows nDCG@10 improving. Your users say the answers got worse. Both are true, and this is one of the best-evidenced findings in the RAG literature.

eRAG found only a small correlation between classical retrieval metrics and downstream answer quality, and improved Kendall's tau by 0.168 to 0.494 by instead evaluating each retrieved document through the downstream LLM itself. UDCG states it more bluntly — nDCG, MAP and MRR "do not accurately predict RAG performance" — and improves correlation with end-to-end quality by up to 36%.

The mechanism explains why this is structural rather than a tuning problem. Classical ranking metrics assume a human reading down a result list, so they apply a positional discount: a correct answer at rank 1 is worth more than at rank 8. An LLM does not read sequentially. It processes all retrieved documents as a whole, so the rank-position discount is measuring something that does not happen. Worse, the metrics assume irrelevant documents are merely neutral — costing you a click. In generation, related-but-irrelevant documents actively degrade the output, which means a retrieval change that raises recall by pulling in more topically-similar-but-wrong passages can lower answer quality while your dashboard turns green.

The practical evaluation stack that follows from this:

• A golden set from your own traffic: Two to three hundred real user queries with human-labelled correct answers, versioned in the repository. This is the acceptance criterion. Everything else is diagnostics.
• End-to-end answer scoring, run in CI: Score the final answer, not the ranked list, on every change to chunking, embeddings, top-k, the reranker, the prompt, or the model. Retrieval and generation changes interact, so they have to be evaluated together.
• Retrieval metrics kept as debugging instruments: Recall@k tells you whether the right passage was available at all — genuinely useful for localizing a failure. It does not tell you whether the answer was good.
• A sufficiency check before generation: Classify whether the retrieved set actually contains enough to answer. Route insufficient cases to abstention or escalation rather than letting the model improvise.
• Frameworks used with open eyes: RAGAS is actively maintained (v0.4.3, January 2026) and useful for fast iteration. Under systematic corpus mutation across more than 28,000 mutants, its best metric reached F1 of 0.570 at fault detection, against 0.927 to 1.000 for a metamorphic oracle. It is a signal, not a gate.
• Vendor leaderboards read as marketing: The most-cited hallucination leaderboard is vendor-run and vendor-scored with the vendor's own model, on summarization only. Its ordering may be right; it is not independent evidence, and it does not describe your task.

If you cannot show a stakeholder a chart of end-to-end answer accuracy on your own labelled queries, moving across dated commits, you do not have an evaluation. You have a dashboard.

— Frenchy Digital evaluation principle

Every one of these is measured, and every one of them will happen to you at some scale. Design for them explicitly rather than discovering them in a stakeholder demo.

Documented RAG failure modes with primary-source evidence, August 2026.

The first row is the one that surprises executives. Correct retrieval does not guarantee a correct answer, and retrieved context is sufficient far less often than teams assume. The Sufficient Context work (ICLR 2025) found that on Musique, 55.4% of instances did not contain sufficient context to answer at all. Its headline result is that state-of-the-art models output correct responses only 35% to 62% of the time when context is insufficient, and its blunt one-line summary is that "with RAG, models hallucinate more than abstain."

The behavioural split in that paper is what you design around. Larger models "excel at answering queries when the context is sufficient, but often output incorrect answers instead of abstaining when the context is not." Smaller models "hallucinate or abstain often, even with sufficient context." So a bigger model does not remove the failure — it relocates it into the cases where your retrieval was weakest, which are exactly the cases nobody is watching.

The most actionable part of that work is the autorater. A sufficient-context classifier reached 93% accuracy (F1 0.935) and — the important detail — requires no ground-truth answer. That means you can measure retrieval sufficiency on live production traffic without a labelled set, which is one of the very few quality signals in this field you can compute continuously. Routing insufficient cases to abstention or escalation improved the fraction of correct answers by 2 to 10%.

The second row is the one that surprises engineers, and it is the cheapest thing here to act on. In the Lost in the Middle experiments, increasing retrieved documents from 20 to 50 raised retriever recall from roughly 69 to roughly 88 — a large improvement in the metric on your retrieval dashboard — while answer accuracy moved by only about 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3. Reader performance saturates long before retriever recall does. You are paying for thirty extra documents of context on every query, on every token, forever, to buy approximately nothing — and the extra passages are disproportionately near-misses, which is the category that actively harms generation. Retrieve shallower, rerank harder, and scope the query to a subcorpus before the semantic search runs.

A note on the "noise helps" literature, because it gets cited to justify large top-k values. A SIGIR 2024 study found that adding random documents could improve accuracy by up to 35%, while highly ranked but irrelevant documents harmed it. A 2026 replication reportedly finds the effect fragile — appearing, weakening or disappearing depending on conditions. Do not design a system around a result that does not reliably reproduce; tune top-k against your own evaluation set and re-tune after every index or model change.

Assembling everything above into the stack we actually build. It is deliberately unexciting; the interesting parts of a retrieval system are the parts that fail an audit or a scale test.

• 1. Governed source of truth: Raw documents, their metadata, their ACLs and their chunking configuration live in a store you control and can rebuild from. The index is always derivative and always disposable. This single decision makes re-embedding, erasure and audit tractable.
• 2. Ingestion with recursive 200-token chunking: Recursive splitting at roughly 200 tokens with modest overlap, per the most efficient configuration in Chroma's evaluation. Skip semantic chunking. Consider late chunking for long technical documents where the measured gains are largest.
• 3. Contextual prefixes at index time: A 50-to-100-token generated summary of the surrounding document prepended to each chunk before embedding, at roughly $1.02 per million document tokens, one time. The prefix goes into both the vector index and the lexical index — a third of the published gain comes from the BM25 side.
• 4. Dual index: lexical and dense: BM25 or a learned-sparse model alongside dense embeddings. On the largest independent benchmark available, the lexical channel alone outperformed the dense one; running both and fusing them outperformed either.
• 5. Permission filtering at query time, failing closed: Late binding against the source system where you can afford the latency; pre-filtering against copied ACLs where you cannot, with propagation lag monitored as an SLO. A missing tenant or ACL predicate returns nothing, never everything.
• 6. Domain scoping before semantic search: Classify or route the query into a subcorpus first, then retrieve shallowly inside it. Depth is not the lever: raising retrieved documents from 20 to 50 lifted retriever recall from roughly 69 to roughly 88 while moving answer accuracy by about a point. Precision in a narrow scope beats depth across everything.
• 7. Fusion, then reranking with a small cross-encoder: Reciprocal rank fusion over the two channels, then a cross-encoder over the top 100 down to the top 10. A 149M-parameter reranker at roughly 150–170ms delivered +20.33pp Hit@1 in independent testing, beating a 4B model that took over a second.
• 8. Sufficiency gate before generation: Classify whether the retrieved set can support an answer. Insufficient means abstain or escalate, not improvise — with RAG, models hallucinate more than they abstain, and are correct only 35% to 62% of the time when context is insufficient. A published autorater hits 93% accuracy without needing ground-truth answers, so this is measurable in production.
• 9. Generation with citations and a bounded tool surface: Every claim traceable to a retrieved chunk. Retrieved content is data, never instruction. Tools allowlisted per workflow, arguments the user never supplied rejected, consequential actions gated on human confirmation.
• 10. Evaluation and tracing in CI: A versioned golden set of a few hundred real queries with labelled answers, scored end to end on every change to chunking, embeddings, top-k, reranker, prompt or model. Full request tracing so any bad answer can be replayed with the exact context it saw.
• 11. Lifecycle: sync, supersession, erasure: Incremental re-indexing on document change, explicit supersession so stale values are not served as current, and an erasure path that reaches the on-disk index through compaction or rebuild, with verification recorded.

The layers teams skip are five, six, eight and eleven — permissions, scoping, the sufficiency gate, and lifecycle. Those four are exactly the ones that turn a working demo into a system that survives its first year, its first data subject access request, and its first thousand documents.

Every one of these has appeared in a real architecture review or vendor evaluation we have run. None are hypothetical.

The Frenchy Digital red-flag list for enterprise RAG proposals, 2026.

Ask for one artifact instead of a deck: an end-to-end accuracy chart on a labelled query set, with the corpus size, the labelling method and the date. A team that cannot produce it has not measured, whatever the architecture diagram shows.

— Frenchy Digital buyer’s principle

These are the bands Frenchy Digital uses to scope retrieval and knowledge-base engagements in 2026. They assume evaluation, permissions and lifecycle are in scope from the start, because retrofitting those is what makes the second year expensive.

Frenchy Digital cost bands for enterprise RAG and knowledge-base engagements, 2026.

Senior-led delivery runs $150 to $225 per hour, and ongoing retainers run $2,500 to $9,500 per month covering model and dependency upgrades, evaluation-set expansion, reindexing runs, incident response and a quarterly technical review. Every engagement carries a 30-day post-launch warranty, and you receive a written scope with a fixed-price phased proposal within 5 business days of the discovery call.

The line item people forget is the running cost of the corpus rather than the queries. Preprocessing with contextual prefixes is roughly a dollar per million document tokens, once. Embeddings run from $0.02 to $0.13 per million tokens depending on the model. But a reindex is a recurring cost, not a one-off: budget at least one full re-embedding per year for model upgrades, corpus restructuring or chunking changes, and price it against the model you chose — the difference between a small and a large embedding model compounds every single time you do it.

Sequencing matters more than tooling. The first workflow pays for the ingestion pipeline, the permission model, the evaluation harness and the tracing. The fourth workflow inherits all of it. Organizations that build one retrieval substrate and add surfaces to it get materially better economics than organizations that pilot four disconnected vendor products in parallel and end up with four permission models to reconcile.

This article is built on the strongest evidence we could verify to primary sources. That evidence is still thinner than the confidence with which this field is usually discussed, and you should know exactly where the soft ground is.

• The hybrid and query-rewriting numbers come from one corpus: 23,088 queries is a large sample, but they are financial documents with substantial tabular content — a setting that flatters lexical matching. Expect the BM25-versus-dense ordering to shift on long narrative prose. The structural conclusion (run both, fuse, rerank) is what transfers.
• The reranker comparison is one narrow domain: 145,000 product reviews, 300 queries, Hit@1 rather than NDCG, and no commercial reranking APIs tested. The size of the effect is large enough to act on; the exact leaderboard ordering is not something to build a procurement decision on.
• Lost in the Middle used 2023-era models: Whether current models have overcome positional degradation is contested, not established. We could not verify the 2026 reproductions circulating in either direction, so we cite neither. Test it on your own workload.
• GraphRAG's own cost evidence is a vendor self-evaluation: LazyGraphRAG's numbers come from Microsoft evaluating Microsoft, on 5,590 articles and 100 synthetic queries scored by an LLM. It is the best signal available that full GraphRAG indexing is over-priced, and it is not independent.
• Several widely quoted figures are unsourced and we omitted them: GraphRAG per-corpus indexing costs, Cohere reranker deltas, Copilot oversharing percentages, and specific counts of exposed vector database instances all trace to marketing or SEO content rather than primary sources. Their absence here is deliberate.
• Embedding leaderboards do not describe your corpus: MTEB-style standings are measured on public benchmark data and are dynamic. The only evidence that describes your retrieval quality is a few hundred labelled queries from your own users.
• Prompt injection has no solution: Nothing in this architecture makes a retrieval agent safe against adversarial content in its own corpus. Everything here reduces how much a successful injection can reach. Treat any vendor claim to the contrary as disqualifying.
• Organizational outcomes are mixed, and the honest numbers are moderate: In a Gartner survey of 782 infrastructure and operations leaders fielded in November and December 2025, only 28% of AI use cases fully succeeded and met ROI expectations, 20% failed outright, and 77% of organizations delivered at least one successful use case. That is materially less pessimistic than the widely repeated failure statistics, and it is real measurement rather than prediction.

The conclusion we would defend is narrow and, we think, useful. Retrieval quality is an engineering problem with measurable answers, and most enterprise RAG systems are leaving large, cheap wins on the table while paying for techniques with evidence against them. Add the lexical channel. Add the small reranker. Add contextual prefixes. Delete HyDE and semantic chunking. Then spend the remaining effort on the part nobody demos — permissions, erasure, evaluation and lifecycle — because that is what determines whether the thing still works in eighteen months.