RAG explained: an AI assistant that answers from your documents
What a RAG chatbot is, when it beats fine-tuning or plain search, and how to test its answers and citations before staff rely on it.
A RAG chatbot answers questions from your own documents instead of from whatever a general language model happens to remember. It finds the passages in your files that relate to the question, hands them to the model, and asks the model to write an answer based on those passages, with a reference to each source. If the documents do not contain the answer, a well-built system says so.
This article explains how that works, when RAG is the right choice compared with fine-tuning or plain search, what usually goes wrong, and what you need to prepare before a project starts. It is written for owners and operations leads who need to make the decision, not for engineers.
What RAG means in plain words
RAG stands for retrieval augmented generation. The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The core idea is simple: a language model that can look up external text before it answers depends less on what it memorised during training.
A company RAG system runs in two phases. First, the documents are prepared. Each file is split into chunks, usually a few paragraphs each, and every chunk is converted into a numerical vector that represents its meaning. The vectors and the original text go into a search index. Second, when a question arrives, the question is converted into a vector too. The index returns the chunks that are closest in meaning, and the language model receives the question together with those chunks. Its instructions say to answer only from the chunks and to name the source for each claim.
Retrieval does most of the work. If the right passage is not retrieved, the model cannot answer correctly, however capable it is. In a typical integration project, much of the effort goes into document preparation and retrieval testing, not into the wording of the prompt.
When is RAG the right tool?
Three approaches get confused. Plain search returns documents or links and leaves the reading to the user. Fine-tuning changes the model itself by training it on examples. RAG keeps the model as it is and supplies the relevant text at question time. The table shows the trade-offs.
| Search box over documents | Fine-tuning | RAG chatbot | |
|---|---|---|---|
| What the user gets | A list of files to open | A model that has learned a style or task | A direct answer with source passages |
| When documents change | Index updates, visible at once | A new training run is needed | Changed files are re-indexed |
| Source shown | The file itself | Usually none | Passage and document reference |
| Good for | Finding a known file | Tone, output format, classification | Questions answered from a defined document set |
| Main risk | Users give up before finding the answer | Confident answers with no source, and stale knowledge | Missed passages, and answers that go beyond the sources |
A simple rule: if the question is “which document covers this”, search is enough. If the question is “what does our policy say about this case”, RAG fits. Fine-tuning teaches a model how to respond. It is a poor place to store facts that change every month. Many teams end up with RAG and a plain search box side by side, because staff sometimes want the original file itself.
Where a company RAG assistant fits
Three cases come up most often.
An internal policy assistant. Employees ask about leave rules, expense limits or the steps of a refund process. The sources are the HR handbook, the finance policy and the procedure manuals. The assistant answers in a sentence or two and links to the section it used, so the employee can read the full text when the answer matters.
A support agent over product documentation. Support staff or customers ask how a feature works, what an error code means, or whether a plan includes an option. The sources are the product documentation, release notes and the known-issues list. The main gain is consistency: the answer comes from the current documentation, not from whichever agent remembers it best.
A sales team assistant over contracts and price lists. A sales representative asks for the standard payment terms of a contract type, or which discount needs approval. The assistant reads the approved price list and the contract templates. This use case only works when access rules are right. A junior representative should not see margins or an old price list.
In each case the assistant is a reading tool for people who make decisions. It does not make the decision for them.
What goes wrong
Most RAG failures do not come from the language model. They come from the data around it.
Stale documents. The index may still contain last year’s price list or a policy that was replaced in March. If nobody owns the update process, the assistant gives the old answer with full confidence and a source that looks valid. The fix is a named owner for each document group, a re-indexing schedule or a trigger when a file changes, and a visible “last updated” date on every source.
Poor chunking. When documents are cut at a fixed length, a table can be split in half, a definition can end up in one chunk while its exception lands in the next, and a heading can be separated from the paragraph it describes. Retrieval then returns a passage that is almost right but misses the condition that changes the answer. Chunking should follow the structure of the document, keeping sections, tables and lists whole wherever possible.
Permissions. A search that ignores who is asking will return passages the user should never see. The rule belongs in the retrieval layer, not in the prompt. Candidate passages must be filtered by the user’s access rights before anything reaches the model, because a model that has read a restricted passage can repeat it.
Hallucinated answers. Even with the right passages, a model can add a detail that is not in them, or cite a source that does not support the sentence. Mitigations include instructions to say “not found in the documents” when the passages do not answer the question, a rule that every claim carries a citation, and a check that the cited text really contains the claim. Some model APIs now return citation references tied to the passages they used. The Anthropic citations documentation describes one such feature, and the principle applies whichever provider you choose.
How do you test a RAG chatbot before launch?
Testing starts before anyone uses the assistant. Collect real questions from support tickets, emails, chat logs and the notes of the sales team. Include the awkward ones: questions with typos, questions that touch two policies at once, and questions the documents cannot answer at all. For each question, write down the correct answer and the document section that contains it. This set is your evaluation set. It is the most useful output of the project, because it stays valid after the model or the vendor changes.
Score two things separately. Answer accuracy asks whether the answer is correct. Citation correctness asks whether the cited passage actually supports the answer. A chatbot can give a correct answer with a wrong citation, and an auditor or a customer who checks the source will count that as a failure.
When an answer is wrong, find out where it failed. If the right passage was not retrieved, the problem is in retrieval or chunking. If the passage was retrieved and the model still answered wrongly, the problem is in the instructions or the model. If the passage was out of date, the problem is in document management. Each failure type has a different fix, and confusing them wastes weeks.
Run the full evaluation set again after every change: a new document, a new chunking rule or a new model version. Keep the results in a simple table, so the team can see whether the system is improving or only changing.
Hosting and data choices
Two hosting paths are common. In the first, the system calls a hosted model API. The model is strong and you do not manage servers, but the retrieved passages, which may contain customer or staff data, are sent to the provider. Contract terms, data retention settings and the processing location must be reviewed before any personal data goes into the index.
In the second, an open-weight model runs on infrastructure you control, either in your own data center or in a cloud region you choose. Open-weight models are often weaker at hard reasoning, but for answering from a defined document set they are frequently adequate, and the data never leaves your environment. The trade-off is operational. Someone has to run the servers, update the model and monitor the results.
For Turkish companies there is also a legal dimension. The company is the data controller, and it is bound by Law No. 6698 on the Protection of Personal Data (KVKK). Article 9 of the law, which covers transfers abroad, was amended in 2024, and a regulation on cross-border transfers followed. The current texts and guidance are published by the Personal Data Protection Authority. Before you choose a hosted API for documents that contain personal data, ask a lawyer to review your position for that specific data. This article is not legal advice.
Language quality is a separate question. Retrieval depends on how well the embedding model understands your language. Turkish uses many suffixes, compound words and abbreviations, so a setup built around English can miss passages that a Turkish reader would find immediately. Test retrieval with Turkish questions and Turkish documents from your own archive, and do not rely on a vendor’s language claims alone.
What to prepare before a project starts
You do not need a finished data platform to begin, but four things are required.
- A document inventory. List every source the assistant may use, with its file type, owner, last update date, language and storage location. Documents nobody owns usually become the stale answers described above.
- Access rules. Write down which people may read which documents. Some of these rules already exist in your file server or CRM. Others will be written down for the first time.
- Example questions. Gather real questions from the people who will use the assistant. Ask what they search for today and what they ask a colleague when they do not know. These become the first version of the evaluation set.
- One owner. A single person decides which documents are authoritative, approves changes to the answer rules and reviews the failure log every month.
If you want to see how this fits your documents and systems, our AI integration service covers the document pipeline, retrieval, access control and the connection to the tools your staff already use. For a first conversation about your documents, contact us.