Context Rot Is Why Your AI Chatbot Gets Worse the Longer a Conversation Runs

Ask a long-running AI assistant the same question twice — once at the start of a conversation, once after an hour of back-and-forth — and you'll often get a worse answer the second time, even though nothing about the question changed and the conversation is nowhere near the model's advertised context limit. Researchers have a name for this: context rot, the measurable decline in a model's output quality as the amount of text in its context window grows, independent of whether that text is actually relevant to the task.

This matters because the entire marketing pitch around modern AI models centers on context size. Vendors advertise windows of 200,000, 1 million, even 10 million tokens, implying that bigger means more capable. Context rot shows the opposite: capability inside that window is not uniform, and most of it degrades well before you hit the ceiling.

What the research actually shows

The clearest demonstration comes from Chroma's 2025 study, which tested 18 frontier models — spanning every major lab — on identical tasks at varying input lengths. Every single model got measurably worse as input length increased, even on tasks explicitly designed to be solvable regardless of context size. The degradation wasn't about the model running out of room; it was about the model's attention diluting across more text.

This builds on the earlier "lost in the middle" finding from Stanford researchers (Liu et al., published in TACL in 2024), which showed a U-shaped reliability curve: models handle information at the very start or very end of a long input well, but information buried in the middle gets used less reliably, regardless of how clearly it's stated. A follow-up evaluation called NoLiMa, run in 2025, found that for many models, the effective context length — the point past which real reliability starts to drop — was around 2,000 tokens, a small fraction of the 100,000+ tokens most of these models claim to handle.

The problem compounds for AI agents that operate over many steps rather than a single question. A June 2026 preprint on long-horizon search found that as an agent's accumulated context grows during a multi-step task, it increasingly tends to give up early or hedge with uncertain answers rather than complete the task — the opposite of what you'd want from a tool meant to work autonomously for longer stretches. The same paper tested seven mitigation strategies across three categories and found that periodic summarization and context pruning — actively removing stale information rather than letting it accumulate — outperformed simply using a model with a bigger window.

Why it happens

The mechanical explanation ties back to how transformer attention works: every token in the context competes for the model's limited attention budget. Add more tokens and each one, on average, gets less of that budget — even tokens that matter. A model is not reading your conversation the way you're reading this article, sequentially with perfect recall; it's re-weighing everything in the window on every single response, and a larger window just means more competition for the same fixed capacity.

Some teams have found the degradation isn't always gradual — a few reports describe models holding steady performance until a length threshold, then dropping sharply, rather than a smooth decline. The research community hasn't settled this point, but both camps agree on the core conclusion: raw context size is a ceiling, not a guarantee, and a lot of real-world usage is already operating past the point of reliable performance well before hitting that ceiling.

What this means if you use AI tools daily

For a day-to-day chatbot user, the practical fix is simple and already within your control: start new conversations more often. If you've been going back and forth with an assistant for 45 minutes across a dozen topic changes, don't assume it still has a clean read on your original question — it's plausibly drowning in its own conversation history. Restating your actual question plainly, rather than relying on the model to dig it out of scrollback, measurably helps.

For anyone building on top of these models — chatbots, internal tools, AI agents handling multi-step workflows — the takeaway is more concrete: don't treat a large context window as a substitute for retrieval and summarization. Dumping an entire knowledge base, codebase, or document set into context because the window technically fits it is exactly the failure mode the research describes. Compacting context periodically, retrieving only what's relevant to the current step, and explicitly re-stating key constraints near the end of a long prompt — where models read more reliably — all measurably outperform relying on sheer window size.

The broader lesson is that "context window" and "usable context" are different numbers, and the gap between them is often much larger than vendors' spec sheets suggest. Treat window size the way you'd treat a hard drive's advertised capacity versus its real-world usable space — real, but not the number that should drive your actual workflow.