What Is RAG? Retrieval-Augmented Generation Explained in Plain English

Photo: Hippopx (public domain). Illustrative image.
RAG stands for retrieval-augmented generation. It is the default architecture behind most enterprise AI in 2026: the chatbot that answers from your company handbook, the support assistant that quotes your help pages, the research tool that cites your papers. The idea is simple. A language model on its own can only answer from what it learned in training — and training ends months before you ask the question. RAG gives it something to look up first: it retrieves the relevant passages from your documents, adds them to the prompt, and then generates an answer grounded in what it just read. This guide explains how it works, when to use it instead of fine-tuning, and where it goes wrong.
Frequently asked questions
How does RAG actually work?
In six steps. Chunk: your documents are split into passages, typically 300–500 tokens each with 10–20% overlap so a fact split across a boundary is not lost. (OpenAI’s vector-store default is 800-token chunks with 400 tokens of overlap.) Embed: each chunk is turned into a vector — a list of numbers that captures its meaning — by an embedding model. Store: the vectors go into a vector database, which can find the nearest neighbours to any new vector fast. Retrieve: when you ask a question, the question is embedded too, and the database returns the chunks closest in meaning. Augment: those chunks are pasted into the model’s prompt, alongside your question. Generate: the model writes an answer using the retrieved passages as its source material. The model never “learns” your documents — it reads them fresh, every time.
What problem does RAG solve?
Two problems. First, stale knowledge: a model trained in January cannot know your prices changed in March, but a RAG system can, because the retrieval step reads the current document. Second, hallucination: a model asked about something it was never taught will often invent a confident answer. Given the right passages in its prompt, it is far more likely to answer from them — and to say it does not know when the passages do not contain the answer. If you want the background on why models invent things, read our guide to AI hallucinations.
Where did RAG come from?
The technique was introduced in a 2020 paper from Facebook AI Research (Lewis and colleagues, NeurIPS 2020), which showed that combining a retriever with a generator beat a generator alone on knowledge-heavy questions. It stayed a research idea until chatbots went mainstream — then it became the standard way to connect a general model to specific knowledge. A 2024 Menlo Ventures survey found 51% of enterprise AI implementations used RAG, up from 31% the year before.
RAG vs fine-tuning: which should I use?
They solve different problems, and they are often used together. Use RAG when the knowledge changes often, lives in your documents, or needs to be cited — company policies, product manuals, legal files, research papers. The knowledge stays in your documents, where you can update or delete it, and the model can show its sources. Use fine-tuning when you want to change how the model behaves — its tone, its format, its skill at a specific task — rather than what it knows. Fine-tuning bakes knowledge into the model’s weights, which makes it hard to update and impossible to cite. The rule of thumb: RAG for knowledge, fine-tuning for behaviour. For the other half of this choice, read our guide to fine-tuning.
What are the variants?
The basic pipeline has several upgrades. Hybrid retrieval combines vector search (meaning) with keyword search (exact terms), so a search for a product code or a person’s name does not get lost. Graph RAG stores the relationships between facts — who works for whom, which part fits which machine — and retrieves along those links, which helps with questions that span several documents. Agentic RAG lets an agent decide what to retrieve, run several searches, check the results and try again, instead of a single fixed retrieval step. Each variant adds complexity; start with the basic pipeline and add a variant only when you can name the failure it fixes.
Where does RAG go wrong?
Most failures are retrieval failures, not generation failures. Bad chunking splits a fact across two chunks, so neither chunk contains the whole answer. Bad retrieval returns passages that are similar in wording but wrong in substance — the classic case is a document that mentions the same words as the question but answers a different one. Too much retrieval floods the prompt with irrelevant passages, and the model gets confused or ignores the good ones. Stale documents are the quietest failure: the system works perfectly, on last year’s prices. The fixes are unglamorous — clean documents, sensible chunk sizes, a small test set of questions with known answers, and a human checking the answers regularly. No amount of model quality compensates for a messy document store.
How do I start?
Start small and concrete. Pick one document set with a clear owner — a help centre, a handbook, a set of product pages. Chunk it, embed it, and build the simplest possible loop: question in, passages out, answer with citations. Test it on twenty questions you already know the answers to, and look at which ones fail. Most will fail in retrieval, not generation, and the fix is usually in the documents: a missing page, a fact buried in a table, a passage split in the wrong place. Only then worry about hybrid search, reranking or agents. And before you hand a RAG system real decisions, run it through our guide to evaluating a new AI tool.
The bottom line
RAG is not magic. It is a filing cabinet attached to a chatbot: the model is only as good as what it can find, and what it can find is only as good as your documents. But it is the architecture that turned chatbots from party tricks into working tools, because it solves the two problems that mattered most — stale knowledge and invented answers — without retraining the model. If you understand chunking, retrieval and the RAG-versus-fine-tuning choice, you understand most of what enterprise AI actually does. For the building blocks, start with our guide to large language models and our guide to AI agents.
Sources
- H2O.ai — “What Is Retrieval-Augmented Generation (RAG)?”: https://www.h2oai.com/wiki/retrieval-augmented-generation
- Lewis, P. et al. — “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (NeurIPS 2020)
- Menlo Ventures — State of Generative AI in the Enterprise, 2024
- FreshSynapse — “What Is Fine-Tuning? A Plain-English Guide to Teaching an AI Model New Tricks”: https://freshsynapse.com/articles/what-is-fine-tuning/
- FreshSynapse — “What Are Large Language Models? A Plain-English Guide”: https://freshsynapse.com/articles/what-are-large-language-models/
- FreshSynapse — “AI Hallucinations: Why Chatbots Make Things Up — and How to Catch Them”: https://freshsynapse.com/articles/ai-hallucinations-explained/
- FreshSynapse — “What Is an AI Agent? A Plain-English Guide to How AI Agents Work”: https://freshsynapse.com/articles/what-is-an-ai-agent/
- FreshSynapse — “How to Evaluate a New AI Tool Before You Trust It”: https://freshsynapse.com/articles/how-to-evaluate-a-new-ai-tool/