What Is RAG?
Retrieval-Augmented Generation Explained

AI models are trained on a snapshot of the internet. Once that training finishes, the model's knowledge is frozen. Ask it about something that happened yesterday, something buried in your company's internal docs, or a niche topic it never encountered during training — and it will either refuse to answer or, worse, confidently make something up.

Retrieval-Augmented Generation — RAG — is the most widely used technique for solving this problem. Instead of relying purely on what the model memorised during training, RAG fetches relevant documents at query time and feeds them directly into the prompt. The model then generates its answer based on that retrieved context.

How RAG Works in Three Steps

1. You ask a question. The system takes your query and converts it into an embedding — a numerical representation that captures its meaning.
2. Relevant documents are retrieved. That embedding is compared against a pre-indexed collection of documents (your knowledge base). The most semantically similar chunks are pulled back — typically 3 to 10 passages.
3. The model generates an answer. Those retrieved chunks are injected into the prompt alongside your original question. The model reads them and produces a grounded response.

The key insight: the model is not searching its memory. It is reading fresh documents that were handed to it moments before answering. This means its response can be based on data the model has never seen during training.

Why RAG Matters

• Reduces hallucination. When the model has actual source material to work from, it is far less likely to fabricate facts. Not impossible — but significantly reduced.
• Keeps answers current. Your knowledge base can be updated daily or even in real time. The model always works with the latest information.
• Works with private data. Company documents, internal wikis, customer records — data that could never be part of a public training set.
• Cheaper than fine-tuning. Fine-tuning a model on your data requires compute, expertise, and maintenance. RAG just requires indexing your documents and adding a retrieval step.

When RAG Works Well

RAG is the right choice when:

• You have a specific knowledge base — product docs, legal contracts, research papers, internal policies.
• Accuracy matters more than creativity. RAG grounds the model in source material.
• Your data changes frequently. Re-index weekly or daily; no retraining needed.
• You need citations. Because the retrieved chunks are known, you can point the user back to the original source.

When RAG Is the Wrong Tool

RAG is not a magic fix for everything:

• General conversation. If you just want a chatbot that talks naturally, RAG adds latency and complexity for no gain.
• Creative writing. You want the model to generate freely, not be constrained by retrieved passages.
• Tiny datasets. If your entire knowledge base fits in a single prompt, just paste it in. No retrieval pipeline needed.
• When the question is about reasoning, not facts. "Solve this maths problem" does not benefit from document retrieval.

Common RAG Pitfalls

Building a RAG system is straightforward. Building a good one is harder.

• Bad chunking. If your documents are split in the wrong places — mid-sentence, mid-paragraph — the retrieved context is incoherent and the model's answer suffers.
• Irrelevant retrieval. The embedding search might return documents that are semantically close but actually off-topic. Garbage in, garbage out.
• Too many chunks. Stuffing 20 passages into the prompt can confuse the model. It may pick up on the wrong one or lose focus.
• Ignoring the retrieval quality. Most teams spend 90% of their effort on the generation side and 10% on retrieval. It should be the opposite.

RAG vs Fine-Tuning vs Long Context