Enterprise RAG: what it is and when it makes sense to use

How it works, without abstraction

The process has two phases. The first happens before any question: the company's documents are split into chunks, converted into numerical representations and stored in a way that allows search by meaning, not just by exact word.

The second happens with every question. The system converts the question the same way, retrieves the chunks closest in meaning and sends them to the model along with the instruction to answer only based on that material.

The most important consequence of this design is that the answer is limited to what was retrieved. If the right chunk is not found, the model cannot get it right — and a well-built system should say it found nothing, instead of improvising.

Quality depends more on retrieval than on the model

It is common to blame a bad answer on the language model. In most RAG failures, the problem comes earlier: the chunks handed to the model did not contain the necessary information.

This happens for very concrete reasons. Documents split into pieces that cut an instruction in half. Internal terms that the search does not associate with what the user asked. Old and new versions of the same document in the knowledge base, with no indication of which one is current. Spreadsheets and tables converted into plain text, losing the structure that gave the numbers their meaning.

That is why working on the document base usually pays off more than switching models. Organizing it, removing obsolete versions and marking what is current has a direct effect on answer quality.

When RAG is not the best answer

Not every question about internal data needs this architecture. Alternatives are worth considering in a few scenarios:

• The information is structured in a database: querying it directly is more accurate and cheaper than retrieving text about it.
• The document set is small and stable: it may fit entirely in the model's context, with no need for search.
• The questions are repetitive and predictable: well-written canned answers solve them more reliably.
• The answer requires calculation or aggregation: language models are not the right tool for that, and RAG does not change this limitation.

What needs to be defined before starting

Some points that, when left for later, tend to cause rework:

• Who can see what: if the knowledge base contains restricted documents, access control must be applied at retrieval, not just in the interface.
• How the knowledge base is updated: documents change, and a system that answers based on a revoked version is worse than not answering.
• What to do when nothing is found: the correct answer is to say it found nothing, and this needs to be explicit in the design.
• Whether the answer cites its source: pointing to the source document lets people verify it, and substantially changes trust in the system.

How to evaluate whether it is working

An impression of quality is not evaluation. The minimum viable approach is a set of real questions with expected answers, defined by people who know the subject, run periodically.

Two things are worth measuring separately: whether the system retrieved the right chunk and whether the generated answer is correct. Separating them shows where to act — if retrieval fails, changing the model will not help.

It is also worth tracking how often the system says it does not know. A very low rate may indicate that it is improvising when it should admit the information is missing.

Frequently asked questions

What is the difference between RAG and training a model on company data?

RAG retrieves information at the moment of the question, so it reflects document changes immediately. Training builds the knowledge into the model, requires repeating the process with every relevant update and is considerably more expensive. For content that changes, RAG is usually the better choice.

Are company documents exposed when using RAG?

It depends on the chosen architecture and on the contracts with the model provider. Access control must be applied at the retrieval stage, ensuring each user only receives chunks from documents they could already read. This decision should be part of the project, not a later adjustment.

Does RAG eliminate the risk of the model making up information?

It reduces it significantly, but does not eliminate it. The model can misread a retrieved chunk or fill in gaps. That is why the explicit instruction to answer only from the material provided, source citation and continuous evaluation with real cases all matter.