When Does AI Need to Read Your Own Files? — An Introduction to RAG

Sometimes you ask an AI a question and get an answer that is perfectly reasonable and completely useless.

Reasonable, because it is generally true. Useless, because it does not know your company, your policies, your products, your files, or the decisions you made last quarter.

That is the gap worth naming early:

A capable model does not know anything about your business on its own.

Ask it "what is our return policy?" and — if it has never read your return policy — it will give you a generic answer, or it will guess.

This is where one idea earns its place: RAG — retrieval-augmented generation.

The idea is simple. Instead of letting the model answer from its general memory alone, you have it read specific sources from your own files first.

The question comes in, the system searches your documents, pulls the passages closest to that question, and hands them to the model so the answer is built on them.

Here is the comparison that makes it concrete.

Imagine you hire someone genuinely bright, on their first day.

They can write, summarise and analyse. But they have not read your internal files. They do not know your pricing. They do not know your support policies. They do not know the questions customers ask every week.

Ask them without giving them anything, and they will guess.

Hand them the policy document and say: "answer from this, and only this" — and the answer gets close to right.

That is the whole job of retrieval.

You need it whenever the answer depends on information that is internal or changes over time: company policies, product manuals, contracts, frequently asked questions, meeting notes, support documentation.

But there is one point worth being honest about:

Retrieval will not turn a mess into a system.

If your files are old, contradictory or scattered, the results will be scattered. If the source is incomplete, the answer will be incomplete. And if the system retrieves the wrong passage, the answer will simply be wrong.

So before asking "how do we build RAG?", ask:

Is our internal knowledge organised? Is there one approved source? Are the files current? Are the headings clear enough to find things by? Does the team know where new information belongs?

There are three levels, and they are worth taking in order.

First: ask the model from its general knowledge. Good for explanations and for thinking out loud.

Second: give it a file inside the conversation. Good for a one-off task, or a single document.

Third: build a knowledge base it can return to. Worth it when you have many documents and the same questions keep coming back.

Do not jump to the third on day one.

The bottom line: If the question is about the world, the model on its own may well be enough.

But if the question is about your files, your policies, your products or your data, it has to be connected to a source.

Retrieval lets the model answer from your knowledge rather than its own — but the quality of the answer is capped by the quality of what it reads.

Organise the knowledge first. Then connect it to AI.