RAG vs Fine-Tuning vs Long Context: How to Choose in 2026

The Short Answer

Retrieve when the corpus is large, changes during the business day, or differs per user. Stuff it into a cached prompt when it is under roughly 200k tokens, stable, and identical for everyone. Fine-tune for format, tone, and latency, never for facts. That third option barely exists on closed APIs now: OpenAI's fine-tuning platform is closed to new users and the Gemini API has no fine-tunable model at all. The real 2026 question is retrieval versus context.

Fine-Tuning Quietly Left the Menu

Every article on this topic still frames it as a three-way choice. For anyone building on a frontier closed model, it is not. OpenAI's supervised and reinforcement fine-tuning guides both carry the same notice: the platform is no longer accessible to new users, and existing users can create training jobs for the coming months. Supervised tuning is capped at gpt-4.1, gpt-4.1-mini and gpt-4.1-nano. Reinforcement tuning is capped at o4-mini. No GPT-5.x model is fine-tunable.

Google went further. Its own model tuning page states that with the deprecation of Gemini 1.5 Flash-001 in May 2025 there is no model supporting fine-tuning in the Gemini API or AI Studio, and that there are no immediate plans to bring it back. Tuning survives only inside the separate Gemini Enterprise Agent Platform.

So "just fine-tune it" now means running open weights on infrastructure you control. Price the hosting floor, not the training run. Amazon Bedrock Custom Model Import bills $0.05718 per Custom Model Unit per minute in us-east-1, and AWS's own worked example states that a customized 8B model with a 128K sequence length requires two units. Multiply that rate out and one unit running continuously costs roughly $2,470 a month, so the two-unit deployment in AWS's own example starts near $4,900 a month before it serves a single token.

The Vendors Selling Million-Token Windows Engineer Against Them

Anthropic's server-side compaction defaults to summarizing the conversation at 150,000 input tokens, on models whose advertised window is 1,000,000 tokens. Anthropic's docs give the reason plainly: as a conversation grows, response quality degrades, so compaction replaces older content with a concise summary. Its context editing tool, once enabled, defaults to firing at 100,000 input tokens and keeping only the three most recent tool use and result pairs.

Its context windows documentation names the phenomenon directly: more context is not automatically better, and as token count grows, accuracy and recall degrade in what it calls context rot. The same page warns that cached prompt prefixes still occupy the window. Caching changes what you pay for those tokens, not whether they count.

NoLiMa (Adobe Research and LMU Munich, ICML 2025) removed literal word overlap between the question and the planted fact, then tested 13 models that all claim 128K or more context. At 32K tokens, 11 of the 13 fell below half of their own sub-1K baseline. GPT-4o dropped from 99.3% to 69.7%. RULER (NVIDIA, COLM 2024) found that of 17 long-context models, only half held satisfactory performance at 32K. Chroma's Context Rot report (July 2025) tested 18 models from Anthropic, OpenAI, Google, and Alibaba, and found on LongMemEval that the full 113k-token prompts underperformed roughly 300-token focused prompts across 306 questions.

Never size a design against the advertised window. Treat anything above 100k input tokens as a region you must measure.

What Long Context Actually Costs

The three vendors price long context on three different principles, and the difference shapes the architecture.

Explicit context caching on Gemini 3.1 Pro carries a storage charge of $4.50 per 1M tokens per hour, so parking a 1M-token cache costs $108 a day before any query runs.

If you are on Claude 4.6 or later, stop using cost as the argument against long context. The remaining arguments are quality, latency, and window pressure. If you are on OpenAI or Gemini, retrieval that keeps every request under the threshold cuts your input bill in half.

Seven Decision Rules

1. Corpus under roughly 200k tokens, stable, identical for every user: do not build RAG. Put it in the prompt behind a cache breakpoint. Anthropic's 1-hour cache writes cost 2x base input and reads cost 0.1x, so its own stated break-even is two reads.
2. Corpus over roughly 1M tokens, or any part of it changes during the business day: retrieve. No caching scheme survives a mutating prefix, and Claude's cache is hierarchical, so a change at the tools level invalidates everything behind it.
3. Different users see different subsets: retrieve, always, and enforce access control in the query filter rather than in the prompt. A stuffed context has no per-row permission model.
4. The model is wrong about facts: retrieve, do not fine-tune. Gekhman et al. (Google Research and Technion, EMNLP 2024) showed that fine-tuning examples carrying new knowledge are learned significantly slower and, as they are learned, linearly increase the model's tendency to hallucinate.
5. The question aggregates over the whole corpus, such as counts, totals, or trends: neither. Have the model write SQL or call an analytics endpoint. Top-k retrieval returns k documents, not a population.
6. You are building an agent, not a single-turn feature: multiply your token forecast by 4 to 15. Anthropic measured agents at about 4x chat token usage and multi-agent systems at about 15x, with token usage alone explaining 80% of performance variance on its BrowseComp evaluation.
7. No strong signal either way: default to a small cached stable prefix plus just-in-time retrieval as a tool call, and hold fine-tuning in reserve.

What Fine-Tuning Is Good For

Format, tone, task-specific decision boundaries, and latency. Not knowledge. The Gekhman paper states the division directly: models mostly acquire factual knowledge through pre-training, and fine-tuning teaches them to use it more efficiently.

If you do fine-tune on OpenAI, check the arithmetic first. Fine-tuned gpt-4.1 inference is $3.00 in and $12.00 out per MTok against a $2.00 and $8.00 base, a 50% markup on every request forever, plus $25.00 per 1M training tokens. Since fine-tuned input costs 1.5x base, your prompt has to shrink by more than a third just to break even on input cost, before the training bill and before every retrain when the data drifts.

When fine-tuning is genuinely right, LoRA is close to free relative to full fine-tuning. Thinking Machines Lab's LoRA Without Regret (September 2025) found LoRA matches full fine-tuning when applied to all layers, at slightly more than two-thirds of the FLOPs per pass.

The Failure Modes That Cost Real Money

Retrieval fails on paraphrase. Testers phrase questions in the document's own words. Real users do not. That is the NoLiMa result arriving in production as healthy retrieval@k dashboards and unhappy customers.

Distractors beat the needle. Chroma found that a single topically related but wrong document reduces accuracy, and that four compound the damage. Enterprise corpora are full of superseded policy versions sitting next to the current one, which is exactly the condition that produces distractors.

Cache breakpoints on volatile content fail silently. Put a timestamp or the incoming user message inside a cached block and the prefix hash never matches. There is no error. Prompts under the per-model minimum, 512 to 4,096 tokens depending on the model, are also silently not cached. You find out on the invoice.

Measuring the wrong half of the pipeline is the most common failure. Teams instrument recall@k, watch it climb, and never measure whether the answer built on top of those chunks was correct. Retrieval metrics and answer metrics are different numbers and they move independently. Stanford RegLab and HAI tested three commercial legal research products whose vendors marketed them as eliminating or avoiding hallucinations, and measured hallucination rates between 17% and 33%. Those products sit on the best-curated legal databases in existence. Retrieval quality did not save them.

Run the Routing Experiment Instead of Picking a Side

LaRA, a benchmark subtitled No Silver Bullet for LC or RAG Routing, built 2,326 test cases across four QA categories and three kinds of naturally occurring long text, evaluated eleven models, and concluded there is no universal winner: the optimal choice depends on parameter size, long-text capability, context length, task type, and the characteristics of the retrieved chunks.

Build 50 to 100 real queries with known answers, run all three configurations, and let the eval decide. The cheap version is Self-Route, from the Google DeepMind and University of Michigan study (EMNLP 2024 industry track): try retrieval first, escalate to full context only when the model reports the retrieved chunks are insufficient.

The framing most posts use is the wrong axis anyway. RAG and long context both answer one question: how the right tokens get into the window. What determines production quality is what you evict, not what you admit. Anthropic's engineering guidance frames context as an attention budget that a model draws down, not a container you fill, since transformers create n squared pairwise relationships for n tokens. It recommends just-in-time retrieval, where the agent holds lightweight identifiers such as file paths or stored queries and loads the content at runtime.

How This Shows Up in Our Work

At SASID we have shipped 7 production AI systems across 5 industries, including a plain-English query engine running over six months with zero bad queries, HIPAA-compliant retrieval over medical records that cut insurance denial appeals from 30 to 60 minutes of specialist time to under 2 minutes, and a platform processing more than 30,000 reviews a day across 200+ locations. In each case the architecture followed from an eval set of real queries rather than from a debate about window sizes.

Get a Free Technical Assessment

If you are choosing between retrieval, long context, and fine-tuning for a real system, we offer a free technical assessment: a 30-minute call about your use case, then a written roadmap within 48 hours covering architecture, cost per request, evaluation, and timeline. There is no obligation and the roadmap is yours to keep. Book at sasid.ai.