A Memory with a Shelf Life
Out of curiosity I once watched where my tokens go. What first looked like a rip-off turns out to be a story about my partner's forgetful memory — and about when calling it a day gets really expensive.
A Glance at the Gauge
Claude and Codex show me at any time how much of my quota I’ve already used up. Honestly, for a long time I only looked when the limit came into view — the way you only notice the fuel gauge once the needle hits red.
The other day I was simply curious. I tracked what happens to my tokens across several sessions. Not just at the end, but from the very first task on. And what I saw first startled me and then taught me a thing or two about agents and the services behind them.
Always the same picture: I start a session, enter the first prompt — and before anything has really happened, 20 or 30 percent of my five-hour limit is already gone. My first thought every time: oh no, what just happened? Did I accidentally dump half the project into the context?
But after that it went on quite gently. Consumption settled down, the next tasks only nibbled at the quota at a normal pace. What I first took for a display glitch or some shady billing turned out, with a bit of thought, to be quite simple to explain.
Not a Display Glitch, but a Memory
I’ve called the agent, elsewhere, a colleague with amnesia: he starts every task from scratch, all knowledge has to be freshly loaded into his context. And that’s true not just for a session, but for every single prompt. The LLM doesn’t remember our session; it’s handed the whole thing again and again, in full, with every prompt.
So that this reading-in doesn’t come up anew with every single prompt, there’s a kind of short-term memory: the prompt cache. As long as the session is running, the system remembers what has already been read in and doesn’t have to process it over and over. That’s exactly why it gets so pleasantly cheap after the expensive start.
But a cache, of course, eventually gets invalidated.
It happened that I had to pause a session because my five-hour quota was used up. The next morning I wanted to finish the task, picked up right where I’d left off — and watched a huge chunk of tokens vanish with the very first prompt. Apparently the cache was no longer usable overnight. The entire session and all the painstakingly built-up knowledge had to be made available as context again. After that everything ran at the usual, cheaper pace again.
The same thing happened when I didn’t have to pause a session at all, but simply wanted to continue it — because it was in such good flow and I could build on all the knowledge gathered. A few hours later, back at the computer, carrying on: same expensive cold start, same reading-in from scratch.
How Long Does the Memory Last?
A reliable, universally valid promise of how long a prompt cache holds is something I haven’t found so far. My observation: overnight it’s practically certainly gone. Interruptions on the same day, by contrast, often seem to be no problem.
That’s not an official figure, and probably not the same for every service either. But it’s enough for a practical rule of thumb: a short break is usually fine. After a longer interruption, I should expect the session no longer to have its context cheaply in short-term memory.
The lesson from it is unspectacular, but it saves noticeable tokens: when I want to call it a day and I’m still in the middle of a task, I don’t just put the session on pause hoping to pick up seamlessly tomorrow.
Instead, before shutting down, I have the session summarize the essentials in a file: where do we stand? What’s the next step? Which decisions have already been made?
The next day I start fresh, load that compact summary — and build on it, instead of expensively reviving the entire previous day’s session.
It’s basically the same idea as with documentation: what counts isn’t the complete history, but compact, easily findable knowledge at the right moment. A small measure with a big effect on token efficiency.