The context window: why a chatbot forgets the start of a long chat

You are forty messages into a chat with an AI assistant. You mention a peanut allergy early on, then plan a whole trip, and then the assistant suggests a peanut sauce. Did it ignore you? Not exactly. It most likely could not see that early message any more. The reason is a limit called the context window.

A chatbot starts every reply from scratch

It feels like you are talking to someone who remembers. Under the hood, a language model does not keep a diary between replies. In a typical chat setup, each time you press send, the app packs up the whole conversation so far (every message from you and every reply) and hands it to the model as one long piece of text. The model reads it from the top and writes the next reply. Some apps add a separate "saved notes" feature on top, but the basic mechanism is: no resend, no memory.

The window has a size

A model can only read a limited amount of text in one go. That limit is the context window. It is counted in tokens, small pieces of text that are roughly a word or part of a word. The window holds everything: your old messages, your new one, any hidden instructions from the app, and the reply the model is writing. As of 2026, windows range from a few thousand tokens to over a million depending on the model, so check the documentation for the one you use.

When a conversation grows past the window, something has to give. Most apps quietly drop the oldest messages first. The model then answers using only what is left, and from its point of view the dropped messages never happened.

Try it: shrink the window

Drag the slider down and watch the oldest rows fade first. Pick a question and see what the toy assistant can answer. It is a simple rule, not a real model: if the message holding the fact is inside the window, it answers; if not, it has nothing to go on. A real model might also guess, and a confident wrong guess is called a hallucination, which is why "it forgot" can look like "it made something up".

What the trimming looks like in code

This Python function does what many chat apps do: walk backwards from the newest message, keep messages while they fit, and stop at the first one that does not.

def fit_to_window(messages, limit):
    kept, used = [], 0
    for msg in reversed(messages):      # walk from the newest message back
        cost = len(msg.split())         # crude stand-in for a token count
        if used + cost > limit:
            break                       # the next-oldest no longer fits: stop
        kept.append(msg)
        used += cost
    return list(reversed(kept)), used

Running it on the conversation from the demo, with a limit of 30 words, prints this:

kept 4 of 8 messages, 28 words
 - Trains. Also I am allergic to peanuts.
 - Noted. Trains are comfortable and the route is scenic.
 - What should I pack for a weekend there?
 - What is my name?
name still visible: False

Notice the allergy survived but the name did not. The window does not care what is important, only what is recent.

The reply needs room too

One detail catches people out: the window is shared between what goes in and what comes out. If a model has a window of 8,000 tokens and you send it 7,900, only about 100 tokens are left for its answer, and a longer reply would be cut off. Many services let you set a separate cap on the reply length for exactly this reason. So a long pasted document does not just risk pushing old messages out; it also squeezes the space the answer can use.

This is also why the same conversation can behave differently in two apps. One may keep the last twenty messages, another may keep as many as fit under a token budget, and a third may summarise. The model is the same kind of thing in each case; what differs is the code that decides what to hand it.

Why not just make the window huge?

Bigger windows exist, but they are not free. Every extra token is more text for the model to process, which costs time and money. Research on long inputs (for example the 2023 paper "Lost in the Middle") has also found that models can use information in the middle of a very long input less reliably than information at the start or end. A fact being technically inside the window does not guarantee it is used well.

Some apps handle a full window by summarising: they ask the model to condense the old messages into a short note and keep that instead. That saves space, but a summary can leave out the detail you cared about.

What this means for you

When a chat gets long, repeat anything that really matters, such as the allergy, close to where you need it. Start a new conversation when you change topic, so old messages do not crowd out new ones. And if you build your own tool on top of a model, remember that managing the window is your job: decide what to keep, what to summarise and what to drop, because the model will only ever answer from what it is handed.