AI that answers from your documents
A good answer starts with the right passage, not a better prompt. RAG is less about the model than about which documents are found, who may see them and how the answer shows where it came from.
The question decides whether RAG is needed
RAG, retrieval-augmented generation, means the system first retrieves relevant passages and then lets the model answer from them. It suits questions where the answer is in documents that change: manuals, contracts, procedures and support replies.
If the answer is a calculation or a status in a system, fetch it directly. If the model needs to learn a language or a format, training is the right path, not RAG. RAG does not fix bad documents either.
Build the chain from document to answer
A common chain is: ingest the document, split it into understandable passages, create embeddings, retrieve candidates, rerank them and let the model answer with sources. Every step can degrade the result, so measure them separately.
I usually start with simple search and add reranking only when I see that the right passage is among the candidates but ends up too far down.
- Parse the document and keep headings, dates and source.
- Split into passages that make sense without the rest of the page.
- Create embeddings and store them with metadata.
- Retrieve more candidates than you think you need, then rerank.
- Let the model answer, cite and say no when the evidence is missing.
Chunking and metadata
A common mistake is splitting every 500 characters. Then half the answer lands in one passage and half in the next. Split at headings and paragraphs instead, and keep a little context about where the passage came from.
Metadata is often more important than the search itself. Version, date, department and permission decide whether a passage may be used at all. An old contract should not beat the new one just because the words are similar.
Permissions cannot live in the prompt
If different users may see different documents, the filter must sit in the search, not in the instruction to the model. Retrieve only passages the user is allowed to read. Then a leaked document cannot show up in an answer.
The same applies to sensitive fields. Mask them before sending to the model if they are not needed for the answer. Log which documents were used, not the full content.
Citations and an honest no
An answer without a source is hard to review. Have the model return short quotes and a link to the document. If the evidence is insufficient, it should say so and suggest who can answer, instead of filling the gap.
Set a threshold for how relevant a passage must be. It is better to say “that is not in the source material” than to mix two documents about roughly the same thing.
Test with questions you actually receive
Collect 30–50 questions from support, sales and operations. Write down which document holds the answer and what a correct answer must contain. Run the same set whenever you change chunking, model or search.
Measure whether the right document was found, whether the answer was correct and whether the source pointed to the right place. Those three catch different failures. A model can produce a correct answer by luck, and that should not count as a hit.
Related services
Read next
Questions and answers
Can we use our own documents without sending them to an external AI?
Yes. Either run the model locally or send only the passages needed for the question. Which fits depends on how sensitive the data is and what quality you need.
How large should the passages be?
There is no universal number. Start from the document structure and test. A passage should make sense without the rest of the page, but not contain several different answers.
Is it enough to upload the documents to ChatGPT?
For a quick test, it can be enough. For a system several people will use, you need permissions, updates, sources and a way to measure quality.