Should you have a long agent session that has all the context you need, or should you aim for short and focused sessions? I think we all already have the gut feeling that short is better and yet we never want to abandon a long session. We usually like the first response from an agent, but the 10th iteration, where it’s running in loops and still making the same mistakes is where it starts to get annoying.

The reason people usually quote against long sessions is “it gets expensive”. Partially true. The models do bill for input and output tokens, and the more context you input, the more expensive the call is. However the main argument against long sessions isn’t economics, but something else. To understand why long sessions are both okay and not okay, we need to understand two mechanisms that are at play in the background: caching and attention.

Caching: the reason long sessions aren’t as expensive as you think

When you look at any diagram of “how LLMs work”, you’ll see these scary looking images: a token going in one side, climbing through eighty-some stacked layers, something coming out the other end.

One token climbing through the layers of a model, meeting attention heads and neurons at every layer, emerging as an output vector

For today, forget all of that. The only thing you need to know is that every token you send gets turned into a set of numbers, and those numbers are what the model actually works with to figure out what comes next.

A sentence split into four tokens, each turning into its own key/value vector of numbers, which get stored for reuse

The model bills for every one of those tokens, in and out. But underneath, there are two more meters running that you usually never look at, cache write and cache read. Here’s what all of that costs on Opus 5, per million tokens:

  • Input: $5.00, for every token the model has to actually work through for the first time
  • Output: $25.00, for every token it generates back (this includes thinking tokens)
  • Cache write (5-minute): $6.25, for storing the current prefix so it can be reused later
  • Cache write (1-hour): $10.00, for storing it on the longer-lived tier instead
  • Cache read: $0.50, for reusing something that’s already sitting in the cache

That last number is the one that matters. If the start of your conversation hasn’t changed since your last message, the vectors behind it haven’t changed either, so nothing forces the model to recompute them. It just reads what’s already stored, at a tenth of the input price. That’s prompt caching. Most of a long conversation genuinely does bill at that cache-read rate, because most of it is being read, not reprocessed.

Except it isn’t only your prompts that get billed this way. An agent doesn’t wait for you to type before it acts. Every tool call it makes, every time it decides what to do next, is its own request to the model, and each one rebills the entire accumulated prefix at that same read rate, not just whatever’s new since the last call.

Calls 1, 2, 5, and 10 of an agent fixing a rounding bug in Sales-Post.Codeunit.al, each one rereading everything before it, ending at a 23,000-token conversation

Ten steps into a session whose final context sits at 23k tokens, you’ve billed something like 119k cache-read tokens across those ten calls. The context grew linearly. The bill grew quadratically.

But let’s look at the number again. 119k tokens at a tenth of input price is roughly the same bill as 12k tokens at full rate. For ten steps of agent work, that’s nothing. Quadratic growth is real but also boringly affordable.

Do remember though that cache isn’t forever. Anthropic offers two lifetimes, five minutes or an hour, and which one you’re on depends on how you’re working. Claude Code defaults its main conversation to the one-hour cache, though subagents still run on the five-minute one. Go through the API directly on usage-based billing, and you default to five minutes too. Either way it refreshes on every hit, so a session you keep working stays warm indefinitely for free. Walk away longer than that, for lunch or a meeting, and you’re paying for the whole prefix again.

Side note for the GitHub Copilot crowd: if you’re on an OpenAI model, the default cache is on a similar clock, five to ten minutes of inactivity, up to an hour in some cases. For supported OpenAI models, VS Code now sets prompt_cache_retention: "24h" on your behalf, which trades a slower cache tier for surviving a lunch break intact.

So: while the cache is warm, long sessions are genuinely manageable. Carry on.

Attention: the reason long sessions are still a bad idea

Cost, then, isn’t the problem. Attention is, but there’s no meter for this one.

Remember that diagram from earlier, the layers the token climbed through? The brown boxes in there are attention heads, dozens of them per layer, each one deciding for itself how much of the surrounding context matters right now. That’s the model’s actual mechanism for attention. Every one of those heads divides one unit of attention across everything in its context, every single time it produces the next token, and none of them can opt out of spending it. That’s what softmax does, for anyone who wants to look it up later.

What that means in practice: every voter’s weight depends on one thing: how much it looks like the answer to the question being asked right now. A token that clearly isn’t the answer gets almost nothing, and two hundred of them together still get almost nothing. What splits the vote is a handful of tokens that each look like they might be it. And a long session is made of exactly those: the same file read three times, the same plan revised twice, the same error pasted back from four tool calls. Nothing about the relevant token changed. It’s sharing the room with its own near-copies.

The same relevant token holding steady against 200 off-topic tokens, then splitting its share three ways against near-duplicates of itself

Position makes this worse before it makes it better. Models attend well to the end of the context, because recent tokens are usually the most predictive of what comes next, and reasonably well to the very start. The middle just sits there. That’s measured on how well models actually retrieve things, not on the attention weights directly, but it shows up consistently. Whatever you told the agent at the top of a long session, your actual objective, is now buried in exactly the part of context that gets the least attention of anyone in the room.

Attention drawn as a curve over the context window, high at both ends, dipping in the middle where your original objective sits

The voting chamber

Here’s the way I’ve started thinking about it. Context isn’t a desk where old information sits quietly until you need it. It’s a voting chamber. Everything inside votes on the next token, every time, weighted by relevance, always summing to one. Nobody sits out completely, most just end up with almost nothing to say.

Three situations this explains better than “the agent forgot”:

You read a file, edit it, read it again, edit it again, read it a third time. Context now holds three versions of that file. Three voters, all confidently answering “what’s actually in this file”, all convinced they’re right, because at the time each was written, each was. The current version has no special standing. It’s one of three votes, slightly favoured by recency.

You ask the agent for an email, and it writes one. You don’t like the subject line, so you fix that. The close feels off, so you fix that too. A few rounds in, everything still leans back toward that first draft’s structure and tone no matter what you correct, because the first email is still sitting in context, complete and confident, the only voter with a full answer to “what does an email look like around here.” Every correction you send is one voice arguing against a voter that showed up with an entire draft behind it. It doesn’t stand a chance. Scrap the thread instead, and rewrite the original prompt with everything you’ve learned since folded in from the start.

An approach you told the agent to abandon is still in the room. “Drop it” wins for a few turns, while it’s freshest. Then both get buried and the approach creeps back, because every vote is a vote on what to write next, and “drop it” proposes nothing. The only voter holding actual lines is the approach you abandoned, so those get written again, slightly disguised.

Right after "drop it", the retraction is loud and the approach fades; a few turns later the retraction has faded out while the approach still holds actual lines to write

Three chamber panels, one per scenario: three file versions, one email draft against its two corrections, one approach description against its retraction, all voting for the next token

The obvious instinct is that the only way to clear the room is to end the session and start over. That’s not quite right. Your harness already retires some voters on its own, Claude Code keeps one running todo list instead of ten stale copies, plan mode edits the plan itself instead of appending revision after revision, and old tool results age out or get replaced with placeholders as the session goes on, exactly the retirement mechanism the read-edit-read example above needs. You do have the tool, but it’s the harness that decides which voter goes and when, not you.

You cannot mark something as irrelevant from inside the conversation. Telling the agent “ignore that email format” doesn’t remove a voter. It adds one. Now there’s a fourth voice in the room, arguing against the third, and the third hasn’t gone anywhere. Whatever gets retired automatically retires on the harness’s schedule, not yours. Anything else still means summarizing what matters yourself, and starting over.

What to do instead

What the harness can’t automate is judgment: which of your abandoned approaches are worth a full paragraph and which aren’t, when a conversation has drifted so far from its actual goal that no more correcting will fix it. That’s still yours to do. Here’s what to do about it.

Write plans, and hand off between them, more often than feels necessary. Break the work into sections, give each section to a fresh session, have it write a handoff for the next one, and start that one clean. A handoff should preserve the eliminated options, not the narrative. “We tried X, it broke Y, so we went with Z instead” is the expensive, hard-won part. Drop it, and the next session just tries X again.

Use subagents for anything that involves reading a lot. Vol.3 has the example: three subagents digging through the base app, the docs, and a specialist in parallel, each one burning its own context on the reading, and handing the main session back a condensed report instead of the transcript. Same benefit as a handoff, just paid before the pollution reaches the main conversation instead of cleaned up afterwards.

Resuming an old session runs into the same problem, and Claude Code will actually warn you about it now:

Claude Code prompting to resume a 6-day-old, 644.6k-token session from a summary instead of in full

Even at the cache-read rate, 644.6k tokens is the equivalent of about 64k tokens at full input price, and that’s not a one-time cost, it’s what every single agent decision from that point on drags along with it. Don’t resume old sessions in full if you can help it. If you do resume one, resume from a summary.

The trade you’re making

The two mechanisms in this post want opposite things. Caching rewards never touching what came before, append only, everything stays warm. Attention rewards removing what’s gone stale, which means rewriting the prefix, and that burns the cache.

So every handoff is a deliberate cache burn, and it’s worth it: you’re spending real money to buy back the one resource in this whole system that has no meter attached to it.

And it’s cheaper than “deliberate cache burn” makes it sound. A handoff costs one fresh cache write at 1.25 times the base rate, but on a small prefix, because the new session starts lean. That’s a rounding error next to what staying unfocused would have cost you instead.