Fine-tuning vs RAG vs prompt engineering: which when

Three techniques can change what an AI model does for you, in order of cost and complexity:

• Prompt engineering means rewriting your instructions until the model produces what you want. You change the question; the model stays the same.
• RAG — short for retrieval-augmented generation — means giving the AI access to your specific documents so it can answer from your information rather than guessing from its training data. You change what the model can see; the model still stays the same.
• Fine-tuning means training a model further on your own examples so its behaviour shifts permanently. You change the model itself.

“Should we fine-tune the model?” is the most expensive question an operator can ask without realising it. The honest answer in 2026 is almost always no — not because fine-tuning is bad, but because the question usually arrives before the cheaper alternatives have been tried.

The three techniques form a ladder. You climb only when the rung below has demonstrably failed. What follows is the decision tree for the operator deciding which one (if any) their team should reach for.

Prompt engineering changes what you ask. The model stays the same; the instructions, examples, and structure of your request improve. This is what most people are doing when they say they’re “using ChatGPT.” Good prompts have a clear role for the model, explicit constraints, worked examples (few-shot prompting), and a defined output format. Setup cost: zero. Iteration cost: minutes per try. Reversible: completely.

RAG changes what the model sees. The model stays the same; before each answer, a retrieval step pulls relevant passages from your own data and pastes them into the prompt. The model is now answering an open-book exam on your binder rather than a closed-book exam on its memory. Setup cost: days to weeks. Per-question cost: fractions of a cent for retrieval plus standard generation. Reversible: you can swap retrievers or turn it off. (For the mechanics, see RAG explained without acronyms.)

Fine-tuning changes the model itself. You take an existing model and continue its training on a curated dataset of input/output pairs — typically thousands to tens of thousands of examples. The model’s weights shift toward the patterns in your data. Setup cost: weeks to months. Maintenance cost: every base-model upgrade requires re-training. Reversible: only by reverting to the un-fine-tuned model.

Two flavours of fine-tuning worth knowing about:

• Full fine-tuning of a frontier proprietary model — OpenAI’s reinforcement fine-tuning, custom enterprise programs at OpenAI and Anthropic. Heavy, expensive, rare. Five figures and up; only worth it when the workload is large and the model genuinely needs to internalise patterns that prompts and retrieval can’t teach.
• LoRA / QLoRA adapters on top of open-weights models (Llama, Mistral, Qwen). A small “adapter” layer is trained while the base model’s weights stay frozen. Two to four figures of cost, runs on a single GPU, swappable per use case. This is what most “we fine-tuned a model” projects in 2026 actually mean.

The ladder, in order:

1. Try prompt engineering first. Always. Reframe the task, add few-shot examples, tighten output constraints, give the model an explicit role. Most things that feel like “the model is dumb” turn out to be “the prompt was underspecified.” A team that hasn’t spent a week on prompt iteration cannot tell whether fine-tuning is needed — they haven’t built a baseline.
2. Climb to RAG when the problem is missing information. If the model is wrong because it doesn’t know your data, your post-cutoff facts, your proprietary context — that’s a retrieval problem, not a training problem. Fine-tuning is the wrong tool for “make the model know this fact.” Use RAG.
3. Climb to fine-tuning when prompt + RAG demonstrably fall short and you have data. Specifically: you need consistent tone or format across thousands of outputs, prompt instructions aren’t holding the line, RAG has nothing to retrieve (because the task is generative, not retrieval), and you have ≥1,000 high-quality input/output examples sitting in a usable format.

If you’ve skipped a rung, you’ve almost certainly mis-spent the budget. The pattern in the post-mortems is consistent: teams that fine-tuned without first exhausting prompt engineering ended up with a model that was different but not better, plus a maintenance burden that compounds with every base-model release.

Two patterns are worth pulling out of those numbers.

First, the cost gap between rungs is roughly an order of magnitude each step. Prompt engineering is effectively free. RAG is hundreds to low-thousands of dollars one-time plus per-call costs. Fine-tuning is thousands to tens of thousands plus ongoing maintenance. Climbing one rung when you didn’t need to is rarely catastrophic; climbing two is.

Second, the reversibility cost compounds the dollar cost. A bad prompt is replaced in a minute. A bad RAG architecture is replaced in a sprint. A bad fine-tune means writing off the training spend, the dataset curation work, and the eval harness — and often re-doing them on the base model you should have started with.

There’s a fourth lever you’ll hear about in the same conversations: tool use or function calling — where the model decides when to call an external function (a database query, a calculator, a web search, an API) and uses the result in its answer. Agent frameworks (LangGraph, AutoGen, Anthropic’s tool-use API, OpenAI’s responses API) sit on top of this.

It’s not on this ladder because it solves a different problem. The three techniques above are about getting better answers. Tool use is about taking actions or fetching live data. You can — and often should — combine them: a prompt-engineered system that uses tools for calculations and RAG for grounding. But picking tool use instead of the three rungs above is a category error; it doesn’t replace any of them.

If your problem is “the model needs to take an action, check a live system, or do exact math” — that’s tool use. Treat it as orthogonal to this decision, not a fourth option on the ladder.

Prompt engineering fails when the prompt has to encode patterns that don’t fit in a few-shot examples block. If your output spec is 8,000 tokens of edge cases and you keep adding more, that’s a signal the prompt has hit its ceiling. Long, brittle prompts also have their own bill of materials — every token re-sent on every call adds up. (See Tokens, context windows, and what they cost for the arithmetic.)

RAG fails when retrieval misses the right passage. The 2026 consensus from vendor write-ups and academic post-mortems is consistent — when RAG goes wrong in production, it’s almost always a retrieval failure (right answer / wrong passage, multi-hop questions, lost context from poor chunking), not a generation failure. RAG also fails when the task is generative and there’s nothing to retrieve — “write me a marketing email in our brand voice” is not a RAG problem.

Fine-tuning fails when the dataset is small, noisy, or unrepresentative of production traffic; the base model upgrades and the fine-tune drifts; or the team can’t measure improvement (no eval harness). Fine-tuning produces a model that’s different; whether it’s better is a question only an eval harness can answer, and most teams don’t build one before training. The most common production outcome of a rushed fine-tune is a model that’s slightly worse on most tasks and slightly better on the narrow training distribution — a net regression.

• If the answer is prompt engineering, your next read isn’t this playbook — it’s the vendor’s own prompt engineering guide. Anthropic and OpenAI both maintain good ones (linked in Sources below). Build a small set of test cases first; iterate against them.
• If the answer is RAG, start at RAG explained without acronyms for the mechanics, then Build a private knowledge base your team can search for the practical setup.
• If the answer is fine-tuning, the first homework isn’t training — it’s building an evaluation harness on your existing prompt-engineered baseline. If you can’t measure whether the fine-tune improved things, the training run is a guess. The OpenAI fine-tuning guide and the Hugging Face PEFT library (both linked) are the right starting points for proprietary and open-weights routes respectively.

The cost framing in Tokens, context windows, and what they cost is the companion to this piece — once you’ve picked the technique, that’s how to budget the workload.