What is a context window? Why AI forgets details in long chats
You work in the same chat for an hour and AI suddenly ignores your agreements. That is down to the context window, a language model’s working memory.
What is a context window? Why AI forgets details in long chats
You work on a report in the same chat for an hour, and suddenly ChatGPT ignores your instruction to always write in the polite form.
Or you paste an 80 page contract, but the answer misses the exact clause you meant. That can feel like the AI is not paying attention, but often the problem lies in the model’s working memory.
Context windows have grown fast, from 9,000 tokens to 1 million tokens in some models. Even so, bigger is not automatically more reliable. After this article, you will understand why that is, and how to handle long chats more smartly.
What is a context window, in plain language?
A context window is the total amount of text a language model can look at at the same time while building an answer. You can think of it as the AI’s working memory.
That working memory is different from the training data. Training data is the text the model once learned patterns from. If you want to go back to the basics, also read what a language model is.
The context window can hold several things at once. Think of your latest question, earlier messages in the same chat, uploaded files and the answer the model is writing at that moment.
A simple comparison is a desk. Only a limited number of papers fit on it. As long as everything stays on the desk, you can work with it. Once the desk is too full, old papers move to the edge or disappear from view.
That is how AI works too. The model does not have unlimited space to carry everything along in one go. So a promise made at the start of a conversation may be used less well in an answer at the end.
It is also important what the context window is not. It is not permanent memory. If you close the chat or start a new conversation, that earlier context is not there by default, unless the tool uses a separate memory feature or project environment.
How big is a context window, and what is a token?
The size of a context window is measured in tokens, not words. Modern models often have well over one hundred thousand to 1 million tokens, but the exact size differs by model, app, API and provider.
A token is not a word. According to OpenAI, a token can be a character, part of a word, a whole word or punctuation. OpenAI’s rule of thumb for English is: 1 token is about 4 characters or three quarters of a word, and 100 tokens is about 75 words.
For Dutch, that is often less favourable. Dutch words are sometimes longer or split up differently. That means the same number of tokens usually holds fewer Dutch words than English words.
The growth has been huge. Anthropic said in 2023 that Claude went from 9,000 to 100,000 tokens, about 75,000 words. Google lists context windows of 1 million tokens or more for many Gemini models, depending on the model.
Bear in mind that the window is shared. Your input counts, but so does the answer the model writes. For some reasoning models, thought steps can also count within that window.
The size in a regular app can differ from the size in the API. It can also vary by subscription. So always check the provider’s own current figures.
| Tokens | Approx. English words | Source |
|---|---|---|
| 100 tokens | About 75 words | OpenAI Help Center |
| 1 paragraph | About 75 words at roughly 100 tokens | OpenAI Help Center |
| 100,000 tokens | About 75,000 words | Anthropic in 2023 |
| 1 million tokens | No fixed word count, this varies by model and language | Google AI for Developers |
Why does AI forget details in a long conversation?
AI forgets details in long chats because the window can fill up, or because information in a large block of text carries less weight. The model’s attention is then spread across many messages, files and instructions.
In a very long conversation, an app may shorten or summarise older messages. That depends on the tool. You may not always see this clearly in the chat, but it can still affect the answer.
Even if everything still fits in the window, things can still go wrong. Research titled Lost in the Middle shows that models use information in the middle of long input less well than information at the start or end.
You will recognise that in practice. You paste in a long document and ask for all the risks. The model neatly mentions the first few points, and also picks up something from the conclusion, but misses an important passage somewhere in the middle.
This can lead to mistakes that sound very convincing. Sometimes the model fills in missing information with something that seems logical, but is not correct. That links to the problem of AI hallucinations.
That is why checking is necessary, especially with contracts, quotes, policy texts and legal documents. Do not only ask for a summary. Also ask where in the document the information can be found.
What is a context window not? It is not a memory feature
A context window is not a memory feature, because it applies within one conversation. A memory feature or project feature can store information between conversations, depending on the tool and your settings.
This difference matters. The context window is temporary working memory. Memory features are meant to reuse preferences or data later. Project files are files you can keep using within a project environment.
Those three things look similar, but they do different jobs.
| Concept | What it is | How long it applies |
|---|---|---|
| Context window | Text the model can read at once in one conversation | During the task or chat |
| Memory feature | Information a tool can keep for later conversations | Between conversations, depending on the tool |
| Training data | Data the model used to learn patterns | For the model itself, not for your separate chat |
Privacy matters here. Everything you paste into the window is processed by the AI provider. So do not simply paste customer data, special personal data or national ID numbers into a free or private account.
If you work with business documents, use an environment with clear agreements about data processing. Also read how to use AI safely in your organisation.
How do you use the context window wisely?
You use the context window better by keeping conversations short and focused. Now that you know what a context window is, you can repeat important rules and work with summaries for long tasks.
The main rule is one task per chat. Do not use the same chat for a quote, a brainstorm, a complaints letter and a contract analysis. That makes the context messy.
Put your most important instructions at the top. Think about audience, tone, length, language and the goal of the task. For a long task, you can repeat those rules later as well.
A good approach is to have the model make an interim summary. Check that summary. Then start a new chat with that summary as the first message.
For long documents, splitting often works better than asking for everything at once. Break the document into parts, or ask about one section at a time. A question like find the passages about warranty terms is stronger than summarise everything.
Also ask for quotes from the document. That way you can check whether the answer really comes from the text. Let the model say if it cannot find something.
- Start a new chat for a new topic.
- Repeat important instructions at every major step.
- Have long chats summarised before you continue.
- Ask for passages and quotes, not just conclusions.
- Always check important answers in the original document.
Which model has the biggest context window, and do you need it?
There are models with 1 million tokens, but for most work tasks, a focused approach matters more than the biggest window. A large window helps, but it does not solve every problem.
A large context window is useful for long contracts, minutes, case files and reports. It lets you provide more text at once. That makes it easier to spot links between pieces of text that are far apart.
Even so, there is still a risk that details will be missed. That is especially true if you ask a broad question. The longer the text, the more important your wording becomes.
If you use AI through an API, a large window can also have downsides. It can be slower and cost more. So do not look only at the size, but also at your task.
| Situation | Is a large window useful? | Approach |
|---|---|---|
| Rewriting a short email | Usually not needed | Give goal, tone and length |
| Reading a quotation file | Often useful | Ask per topic for passages and quotes |
| Comparing minutes | Useful | Have decisions and open points searched separately |
| Analysing a long contract | Useful, but checking remains necessary | Ask for clauses with page numbers |
So do not rely only on the biggest numbers. Always check what context size your model, app or API has at that moment. The provider gives the current information.
What about privacy and rules in the Netherlands and Belgium?
What you put into the context window is shared with the AI tool provider. That means the GDPR still applies in the Netherlands and Belgium.
That is especially important for customer data, staff information, medical data and financial data. Do not just paste such information into a private account or free environment. Anonymise names, addresses, national ID numbers and other traceable data where you can.
For business documents, a business environment is often easier to manage. Then you can make agreements about data processing. Think about who has access, what is stored and what the provider may use the data for.
The AI Act also requires AI literacy in organisations that use AI. That means staff need to understand what AI can and cannot do. Knowledge of context windows is part of that.
Large AI tools can work well with Dutch. Even so, Dutch text can often be less token efficient than English. Keep that in mind with long passages.
Practical example: Sanne refreshes a long chat
You refresh a long chat by first having the agreements summarised, checking that summary and then starting again. That makes the task clearer for you and for the model.
In this example, Sanne, a project manager at a Dutch installation company, is working with a quotation file of 60 pages. She uses ChatGPT or Claude and had earlier agreed: write formally, in the polite form, maximum 150 words.
After many messages, things still go wrong. The model suddenly writes in the informal form. That does not automatically mean the tool is broken. The chat has probably become too long or too messy.
Sanne handles it like this:
- At the end of the long chat, she types: Make a summary of our agreements and the main decisions so far in no more than 10 points, including tone and length requirements.
- She checks the points and corrects mistakes.
- She opens a new chat and pastes the summary as the first message.
- She adds: These are the starting points for this task. Stick to them and briefly repeat them in your own words before we begin.
- She uploads the file and asks about one topic only, for example warranty terms.
Then she asks: Find the passages in the document about warranty terms and quote them exactly with page number. Say if you cannot find them.
The expected result is that the model again follows the tone and length more closely. Sanne can also check the quotes in the PDF. That remains necessary, because AI can still miss or misunderstand something.
Common mistakes in long chats
The biggest mistake in long chats is trying to do everything in one endless conversation. The fix is simple: work per task in a new chat and carry over a checked summary.
A second mistake is thinking a large context window is the same as remembering everything. It is not. Ask focused questions and have the model give quotes from the document.
A third mistake is confusing the context window with memory or training. The window belongs to one conversation. Memory features and project files work differently and should be used consciously.
A fourth mistake is pasting in a whole file including personal data. Anonymise first and use a business environment with clear agreements for work.
A fifth mistake is only putting important instructions at the beginning. For a long task, it is better to repeat the main rules at the end or mention them again at each step.
Checklist: is my chat too full?
Your chat may be too full or too messy if you see one or more of these signs:
- The model ignores earlier agreements.
- The model repeats itself.
- The model mixes up topics.
- The chat has dozens of messages or long attachments.
- You keep having to explain what had already been agreed.
Frequently asked questions
What is a context window in simple words?
It is AI’s working memory: the text the model can read at the same time. That includes your question, earlier messages, attachments and the answer being produced.
Is a larger context window always better?
No. A larger window can be useful for long documents, but the model can still miss details in long texts. Focused questions remain important.
Is the context window the same as memory in ChatGPT?
No. The context window applies in one conversation. Memory features store data between conversations, depending on the tool and your settings.
How many words fit in a context window?
As a rough guide, count about 3 words per 4 tokens for English. In Dutch, usually a little less fits, because text is often split into tokens differently. So a token count is not the same as a word count.
How do I know my chat is too long?
If the model ignores agreements, repeats itself or mixes things up, the chat is probably too full or too messy. Then start a new chat with a checked summary.
Take a long chat you already have, use the prompt from this article to turn it into a start brief, and test it in a new conversation. If someone then asks you what a context window is, you can explain it as AI’s working memory. If you want to write better prompts next, read on about a good prompt.
Was this article helpful?
