What Is RAG? Do You Still Need a RAG System?

RAG means the AI checks your documents first and answers second. Without RAG it answers from its general training and guesses at everything that only exists inside your company. Two years ago this was a topic for developers. Today managing directors ask whether they need a RAG system. At the same time the technology has moved so fast that the honest answer is no longer an automatic yes.

What is RAG? Explained simply

RAG stands for Retrieval Augmented Generation. In plain words, the answer gets enriched with information that was looked up for exactly this question.

Picture an exam. Without a cheat sheet, the AI writes down whatever sounds plausible. With a cheat sheet, it looks things up and answers with what is written there. RAG is the mechanism that picks the right cheat sheet for every question: from your manuals, contracts, tickets or wiki.

It works in three steps. Someone asks a question. The system searches your documents for the passages that match. The AI gets the question and the passages together and writes the answer, ideally with a source.

What do you need a RAG system for in a company?

The classic example is an internal knowledge base. A new colleague asks: "How do we handle a complaint?" Instead of asking three people or digging through the wiki, she gets the answer from the current process document, with a link to it.

The same principle sits behind a RAG chatbot in support that answers customer questions from your product documentation. Or behind a contract check that compares new contracts with your standard clauses. Everywhere the same rule applies. The AI does not know your knowledge by itself. It has to get it at the right moment.

Doesn't everything just fit into the context window?

Two years ago the answer was a clear no. Today current models like Claude handle up to one million tokens in a single request (Anthropic, as of November 2026 (opens in new tab)). That is a whole shelf of manuals at once.

For a handful of documents that is enough. Then you don't need a RAG system and simply upload the files. With a real company archive you hit limits fast:

• Thousands of documents still don't fit into one window.
• Every request with a huge context costs money and takes longer.
• Models miss details that sit in the middle of a long text. Researchers from Stanford called this "Lost in the Middle" (Liu et al., TACL 2024 (opens in new tab)), and Anthropic itself warns that accuracy drops with very long context.
• Access rights get lost. If you dump everything into the context, the AI also sees what the person asking is not allowed to see.

The real problem isn't the technology, it's your data

Many people think more is better with RAG. Throw in every document there is. That is exactly what goes wrong.

A typical case: a company wants to connect 3,000 PDFs to an AI. Manuals, policies, product sheets, grown over years. My first question was not which database to use. It was: who can tell which of these documents are still up to date? You can guess the answer.

If three versions of the same policy sit in the archive, the search finds all three. The AI then answers with the outdated one, convincingly and with a source. Garbage in, hallucination out.

Preparing your data therefore means:

• removing outdated files and duplicates
• deciding which source wins when two disagree
• sorting out access rights before the AI sees anything
• naming one person who owns each topic

That is not a technical task. It is work on your knowledge, and it decides the quality of the answers more than any model does.

RAG vs. fine-tuning vs. knowledge base

These three terms often show up in the same sentence, yet they solve different problems.

The rule of thumb: fine-tuning changes how the AI writes. RAG changes what it knows. For company knowledge, fine-tuning is almost never the right choice, because every new document needs new training.

Do you still need a RAG system now that there are MCP and APIs?

The principle, yes. The classic way of building it, not always. Two years ago RAG almost automatically meant cutting documents into pieces, loading them into a vector database and building your own search. Today there are three routes.

• Small, stable knowledge: A project in ChatGPT, Claude or Copilot with uploaded documents is often enough. The search is already built in.
• Live data from your systems: Stock levels, open tickets, customer data in the CRM. Here an AI agent fetches the information directly from the system via MCP or an API. A document index is out of date before it is finished.
• A large, mixed document archive with access rights: This is where your own RAG system pays off, connected to your file storage and permissions.

In practice it is often a mix. Which one depends on the use case and your systems. More important than the choice of technology is something else: you need a loop that makes the knowledge better. When someone spots a wrong answer, they flag it with one click. Someone checks the source, fixes or removes it, and the next answer is right. Without this feedback loop a RAG system gets worse every month, because the world changes and the documents don't.

When a RAG setup pays off

If you want to bring AI into real processes, such as support, an internal knowledge base or contract review, you can't get around the principle. The AI needs the right knowledge at the right moment. How that is built technically is the second question.

A good RAG setup does not come off the shelf. It is systems integration: your file storage, your permissions, your processes and a way to keep the knowledge current. That is exactly where our AI consulting comes in. Not traditional consulting with slide decks. We look at your data, tell you which route fits and build it.