RAG: how a chatbot looks things up before it answers

Ask a chatbot "Can I bring my dog to Pebble Café?" and it has a problem. Pebble Café is a small shop, and the chatbot has never seen its rules. It might guess. A guess that sounds confident and is wrong is called a hallucination: text that reads well but isn't backed by anything true.

One common fix is to stop asking the chatbot to remember, and start handing it the answer sheet. This trick is called retrieval-augmented generation, or RAG for short. The name comes from a 2020 research paper, and the idea is simple: look up the relevant notes first, then generate an answer from them.

The model only knows what it was trained on, plus what you paste in

A language model learns from a huge pile of text during training, and then it stops learning. Nothing about your shop, your company's handbook, or last week's price list is in there. But a model can read whatever you put in its prompt, the text you send it (the earlier article on context windows explains how much fits). So instead of hoping it knows, you paste the facts into the prompt.

Pasting your whole handbook every time is wasteful and may not fit. So RAG adds one step before the model sees anything:

1. Split your documents into small pieces called chunks (a paragraph, a few sentences).
2. Retrieve: when a question arrives, score every chunk for how well it matches, and keep the best few.
3. Generate: build a prompt from those chunks plus the question, and send that to the model.

Try the retrieval step yourself

Below are six notes about the café. Type a question, or tap a preset. Each note gets a score, the top ones are highlighted, and the box at the bottom shows the exact prompt the model would receive.

What the scoring is doing

This page uses the simplest possible scoring. It lowercases the text, throws away filler words (can, the, my), trims a trailing "s" so "dogs" and "dog" match, and counts the words the question and the note share. Here is the same idea in Python:

import re

STOP = {"a","an","the","is","are","do","does","can","i","we","you","my","to",
        "of","in","on","at","for","and","or","it","me","with","by","be","how",
        "what","when","where","there","your","our"}

def words(text):
    out = set()
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        if w in STOP:
            continue
        if len(w) > 3 and w.endswith("s"):
            w = w[:-1]
        out.add(w)
    return out

def retrieve(question, notes, k=2):
    q = words(question)
    scored = [(len(q & words(n)), i) for i, n in enumerate(notes)]
    scored.sort(key=lambda s: (-s[0], s[1]))
    return [(score, i) for score, i in scored[:k] if score > 0]

Running it on the six café notes gives this real output:

Can I bring my dog? -> [(1, 1)]
When do you close on Sunday? -> [(2, 0)]
Can I bring my pooch? -> []

Each pair is (score, note number), counting notes from 0. So the dog question picks note 1, the dogs-on-the-terrace note. The Sunday question picks note 0 with a score of 2, because "close" and "sunday" both appear there. And "pooch" finds nothing at all.

Where keyword matching breaks

A person knows that "pooch" and "dog" mean the same thing. The code above only compares spelling, so it doesn't. That is the main weakness of keyword retrieval, and it is why many real systems score chunks with embeddings instead: lists of numbers that place texts with similar meaning close together, so "pooch" lands near "dog". Embeddings have their own misses, but they handle different wording far better than counting shared words.

The rest of a real RAG system is made of the same three steps. Real chunks are longer, there are thousands of them instead of six, and the lookup runs in a specialised index so it stays fast. The shape does not change: split, retrieve, generate.

Why chunks and not whole documents

You might ask why we cut documents up at all. Two reasons. A prompt has a size limit, so a long handbook may not fit. And even when it fits, a prompt full of unrelated text makes it harder for the model to find the one sentence that matters, and every extra word costs time and money. Small chunks let you send only what the question needs. The trade-off is that a chunk cut in the wrong place can lose its meaning: a note that says "it closes at 17:00" is useless if the part naming what closes stayed behind in the previous chunk.

What RAG fixes, and what it doesn't

RAG helps in three ways. The answer can use facts the model never saw in training. When your notes change, you edit a note instead of retraining anything. And you can show the reader which notes the answer came from, so they can check it.

It is not a guarantee, though. If retrieval picks the wrong notes, the model answers from the wrong notes. If it finds nothing, the model is back to guessing unless the prompt tells it to say "I don't know", and even then a model can slip. Two choices matter in practice: how you cut chunks (too small loses context, too big wastes the prompt), and how many you keep. Pick the "group" question above and set k to 3: the booking note is the right one, but a note about dogs and a note about opening hours slip in because they each share one word ("come", "Sunday") with the question.

Take it into your own code

If you ever build a chatbot over your own documents, spend your first hour on retrieval, not on the model. Print the retrieved notes for ten real questions and read them. If the right note is not in the list, no model can answer well, and tuning the prompt will not help. Most "the AI got it wrong" bugs in a RAG system turn out to be "the lookup got it wrong".