How it works
A language model only knows what was in its training data, and it does not know your files. Asked about your refund policy, it will guess. RAG fixes the missing knowledge without retraining the model. Before the model answers, a search step finds the passages most related to the question and adds them to the prompt. The model then answers from that text.
Setting it up has two halves. First you prepare the documents: split them into chunks, turn each chunk into an embedding, and store the embeddings in a vector database. Then, for every question, you turn the question into an embedding, fetch the closest chunks, and send them to the model with an instruction to answer only from them.
Creators meet RAG whenever a tool says it can chat with your documents: a support bot that answers from a help center, a course assistant that answers from lesson transcripts, or a notes app that answers from your own notes. The quality of the answer depends mostly on the retrieval. If the right passage is not fetched, the model cannot use it, and it may still answer confidently.
A worked example
A course creator wants a bot that answers student questions from 60 lesson transcripts.
- Each transcript is split into chunks of a few paragraphs, and an embedding of each chunk is stored.
- A student asks how to export a file. The question is embedded and the five closest chunks are fetched.
- Those chunks go to the model with the instruction: answer from this text only, and say so if the answer is not there.
- The reply names the lesson it came from, so that the student can check it.
An illustration, not a real account. Any numbers in it are made up to show the method, and are not benchmarks.
Common mistakes
- Blaming the model for a retrieval problem. When an answer is wrong, look first at which chunks were fetched.
- Assuming RAG stops made-up answers. It reduces them. A model can still misread a passage or answer when nothing relevant was found.
- Confusing RAG with fine-tuning. RAG gives the model text to read at question time. Fine-tuning changes the model itself.
Questions people ask
When should I use RAG instead of fine-tuning?
Use RAG when the model needs facts that change or that live in your documents. Fine-tuning suits teaching a style or a fixed format. Many projects that seem to need fine-tuning only need better retrieval.
Do I need a vector database for RAG?
For a few documents, no: you can paste them into the prompt or use keyword search. A vector database earns its place when there is too much text to send every time.
Where does the term RAG come from?
From a research paper titled Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. It is linked in the sources on this page.
Sources
The research paper that introduced the term retrieval-augmented generation.
arXiv (Patrick Lewis and others): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks 22 May 2020
OpenAI's guide describes supplying outside or proprietary data within the prompt as retrieval-augmented generation.
Every link was opened on 13 October 2026. Products and policies change, so check the source before you rely on a figure.