In One Sentence

RAG (Retrieval-Augmented Generation) is an architecture that retrieves relevant information from external documents or databases before an LLM generates an answer, then uses that information as evidence.

Think of it as a library: a librarian (the retrieval system) finds the books related to a question and places them on the desk, while an expert (the LLM) reads them before answering. Because the expert does not rely only on memory, the system can answer questions about internal documents and current information.

Why It Matters

LLMs have two structural weaknesses that matter in business use.

First, they only know information available in their training data. A company policy updated yesterday or a price list published this month is not automatically inside the model.

Second, an LLM can produce a plausible answer even when it does not know the truth. This is hallucination: an unsupported answer written in a confident, factual style.

RAG mitigates both problems. It shifts the basis of an answer from the model's memory to actual documents retrieved at query time. This enables answers grounded in internal and current information, and can also make it possible to show the source documents behind an answer.

How It Works

RAG has two broad phases: preparation and answering.

Preparation: Building the Index

Internal documents are first converted into a searchable form. Each document is split into appropriately sized pieces called chunks. An embedding model then converts each chunk into a numerical vector representing its meaning, and the vectors are stored in a vector database.

At Answer Time

When a user asks a question, the system typically:

  1. Converts the question into a vector
  2. Searches the vector database for chunks semantically close to the question
  3. Passes the retrieved chunks to the LLM together with the question
  4. Has the LLM generate an answer grounded in those documents

The key point is that RAG is not one standalone AI technology. It is an architecture combining a retrieval system and an LLM. Quality depends not only on the LLM, but also on document splitting, retrieval accuracy, and the quality and maintenance of the source documents.

When to Use It

RAG is a good fit when the problem is about knowledge.

  • Build an AI assistant that answers from policies, manuals, meeting notes, or past proposals
  • Work with information that changes frequently, such as prices, inventory, or policies
  • Provide sources or evidence with each answer
  • Use company information without the data or budget required for fine-tuning

One major practical advantage is that RAG handles updates well. Replacing or re-indexing a document changes the information available to the system without retraining the model.

When Not to Use It—and Why It Fails

RAG is often misunderstood as "just upload the internal documents." Many low-quality implementations fail because of problems upstream of the retrieval system itself.

  • The documents are not maintained: old and new policy versions are mixed, unwritten rules remain verbal, or important information is buried in tables and diagrams
  • The real problem is behavior: if you need to change writing style, output format, or task accuracy, use SFT or LoRA; RAG alone is not designed for that
  • Cross-document aggregation or reasoning is required: analyzing trends across three years of meeting minutes is difficult for a basic RAG setup
  • There is no evaluation process: without representative questions and expected answers, accuracy cannot be measured or improved

In practice, a RAG project is a knowledge-management project before it is an AI project. Misjudging this often leaves a system stuck at the proof-of-concept stage.

RAG vs. SFT vs. LoRA

The roles of the three terms can be summarized as follows.

Approach Position Problem it addresses
RAG An architecture combining retrieval and an LLM Knowledge, such as internal and current information
SFT A training process using supervised examples Behavior, such as format, accuracy, and style
LoRA A training technique that makes SFT more efficient The same goal as SFT with less compute

RAG and fine-tuning (SFT or LoRA) are not competing answers to the same problem. They address different needs and are often combined: RAG provides knowledge, while fine-tuning shapes behavior.

Use Cases for Small and Mid-Sized Companies

  • Internal help desk: answer questions about employment rules, expense policies, and system manuals to reduce the load on HR and IT
  • Customer support: generate source-backed draft answers from product manuals and FAQs for support agents
  • Sales: find similar projects in past proposals and case studies to gather material for a new proposal
  • Manufacturing and construction: search technical standards and past defect reports to speed up design and inspection work

Each case depends on having source documents that contain the answer. RAG cannot recover undocumented tacit knowledge.

References

  • Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401
  • Gao, Y. et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997

Our Perspective

RAG is a standard architecture for connecting an LLM to external knowledge and answering from internal or current information. Because documents can be updated without retraining the model, and sources can be shown for verification, it is often the first architecture worth evaluating for enterprise LLM use.

It is not, however, a system that works simply by dropping in documents. Start with a narrow business task and representative questions, clean and organize the source documents, prepare an evaluation set, and validate a small implementation. Distinguishing knowledge problems (RAG) from behavior problems (SFT or LoRA) is the first step toward avoiding a poor technology choice.