Language models such as GPT or Llama know a lot about the world, but nothing about your company. They have never seen your product data sheets or your internal policies. Retrieval-augmented generation, or RAG for short, closes that gap. This post explains how it works without the jargon.
How RAG works in three steps
- Retrieve: When someone asks a question, the system first searches your documents for relevant passages.
- Augment: The passages it finds are passed to the language model together with the question.
- Generate: The model writes an answer based on those passages and can cite its sources.
The model does not need to be retrained for this. When your documents change, you simply update the knowledge base.
How does the system find the right passages?
The documents are split into short chunks in advance. For each chunk, an embedding model calculates a vector of numbers that represents its meaning. These vectors are stored in a vector database. When a question comes in, it is turned into a vector as well and compared with the stored ones. This lets the system find passages with a similar meaning, even when different words are used.
Good systems combine this semantic search with classic keyword search. That helps with part numbers, names and technical terms.
Typical use cases for a RAG chatbot
- Internal knowledge assistant: answers about policies, processes and manuals.
- Customer service: a chatbot answers product questions based on your documentation.
- Sales and engineering: quick access to data sheets, quotes and project reports.
What a good RAG system needs
Clean data. Outdated or contradictory documents lead to wrong answers. Decide which sources are authoritative.
Careful preparation. How documents are split determines the quality of the answers. Tables, scans and PDFs with complex layouts need special attention.
Permissions. The chatbot may only use content the person asking is allowed to see.
Measurable quality. A set of typical test questions with expected answers shows whether changes make the system better or worse.
Limitations
RAG significantly reduces made-up answers, but it does not eliminate them. Important answers should therefore come with sources and be checked when in doubt. Questions that require analysis across many documents, such as "How has our revenue developed?", are a poor fit for a pure RAG system. AI agents that query databases directly are better suited to those.
How DFT IT can help
We develop RAG systems and chatbots that work with your data in a GDPR-compliant way. Learn more on our AI development page. Our post on Open WebUI for business shows how a RAG chatbot fits into your own AI platform, and you will find current courses in our AI training overview. Get in touch if you would like to explore a use case.