"Tokens, context windows and why your AI forgets things"

Here is a question I get asked a lot: "why does the chatbot forget what we told it five minutes ago?" A founder shows me their AI-powered support tool, the conversation runs long, and suddenly the assistant has no idea the customer already gave their order number. It looks like a bug. It is usually not a bug. It is a limit built into how these models work, and understanding it will save you money and disappointment.

Start with the token. A token is the unit these models actually read and write, a small chunk of text, roughly three-quarters of a word in English. The word "understanding" might be two or three tokens; "cat" is one. Every AI provider charges by the token, both for the text you send in and the text that comes back. When you hear that a model call cost a certain amount, that number came from counting tokens.

Now the context window. This is the model's working memory: the total amount of text it can see at one time, measured in tokens. Think of it as a whiteboard of fixed size. Everything the model knows about your specific request, your instructions, the customer's question, the documents you gave it, the conversation so far, has to fit on that whiteboard. If it is not on the whiteboard, the model cannot see it. Full stop.

Here is the part that surprises most people: the model has no memory between requests. None. Each time your product calls the model, it starts completely blank, reads whatever is on the whiteboard, writes an answer, and forgets everything. There is no little brain somewhere that remembers your customer from yesterday, or from five minutes ago.

So how does a chat product appear to remember the conversation? Simple: every time the customer sends a new message, the software sends the entire conversation back to the model, all of it, again, plus the new message. The "memory" is a trick, replayed on every turn. And because the whiteboard is a fixed size, a long conversation eventually will not fit. At that point the software has to cut something, usually the oldest messages. That is the moment your assistant "forgets" the order number from earlier. Nothing malfunctioned. The whiteboard filled up.

This has two direct consequences for anyone paying the bill. First, cost: because the whole conversation is re-sent on each turn, a long conversation costs more per message, not less. Turn fifty costs far more than turn two. Second, quality: models pay less attention to material buried in the middle of a very full window. More text in is not automatically better answers out.

You may have heard that context windows are getting huge, the biggest models now advertise windows of a million tokens or more, enough to hold several long novels at once. True, and helpful, but not a cure. A bigger whiteboard costs more to fill on every single call, and independent testing keeps finding that models use very large windows less reliably than the headline number suggests, recall gets patchy well before the whiteboard is actually full. And any business with real history, years of customer records, thousands of documents, will outgrow any window ever made.

The practical answer is to stop trying to fit everything in and start choosing what the model sees. Store your facts in an ordinary database, and for each request fetch only the relevant few pieces and put those on the whiteboard. This approach has a name, retrieval (looking up the right information at the moment it is needed), and it is how nearly every serious AI product works. The skill is not stuffing the window; it is deciding what earns a place on it.

If you are scoping an AI product, three questions will keep you grounded. What exactly goes into the window on each call? What happens when a conversation or a document does not fit? And what does a typical interaction cost in tokens? If whoever is building it can answer those three crisply, you are in good hands.

Designing what a model sees, and keeping the bill sane while doing it, is a large part of the work I do for clients. If you are planning an AI feature and want the token maths done before you commit, get in touch via the contact form.