Slack turned into the hub’s biggest prompt-cache writer this week while reading back only a fifth of what it wrote — 0.22:1 read:write on a single dense session, 72 turns, 3.38M tokens written and 0.75M read back. That’s not the sparse-traffic problem I’d already fixed for Telegram, where gaps between turns are long enough to just expire the cache normally. Slack’s turns were dense, well inside the 5-minute TTL. The cache should have been hitting almost every time. It wasn’t, and that gap between “should” and “was” is the whole story.
| Day | Cache reads | Cache writes | Read:write |
|---|---|---|---|
| Jul 30 | 3.44M | 6.09M | 0.57:1 |
| Aug 4 | 0.88M | 0.70M | 1.26:1 |
| Aug 7 | 1.08M | 3.54M | 0.31:1 |
| Aug 8 | 0.75M | 3.38M | 0.22:1 |
First hypothesis: it’s the tool ordering. I’d already fixed a version of this for voice — a fix that byte-stabilized the sort order of the tools array so a constant set of tools always serialized the same way. Made sense to check first. It wasn’t the problem: Slack was already going through the same ordering fix, and the system prompt’s cached/dynamic split looked clean too. The tools array was sorted consistently. It just wasn’t the same tools from turn to turn.
The breakthrough: the membership itself was churning, not the order. Anthropic’s own docs are explicit that
Prompt caching references the entire prompt – tools, system, and messages (in that order) up to and including the block designated with cache_control.
Tools sit before system in that hierarchy, so if the tools array itself gains or drops entries turn to turn, the whole downstream prefix invalidates no matter how stable everything after it is. And ours was gaining entries, forever.
The actual bug was a function called mergeRuntimeToolGrantSet — a pure monotonic Set union. Every time a conversation expanded its intent or called request_tool_group, whatever tools got granted went into that Set and stayed there. No cap, no TTL, no eviction. Every subsequent request re-seeded all of them as BYPASS patterns exempt from the normal role/profile filtering, so the tool pool for one long-running Slack DM just grew, turn after turn, until it blew past the 100-tool request cap and a prioritization step started slicing the array down — differently, each time, depending on what the current message happened to reference. One DM I measured was pinned at 124 accumulated grants on every request, feeding a pool of 169–345 candidate tools that got sliced to fit. Different slice, different array, different cache key, every single turn.
It’s the same failure shape other people have hit from a completely different angle — building a tools list from something without a stable order (a Python set, an old dict) shuffles the serialized bytes between runs and quietly doubles your cache-write bill. Same underlying rule, two different ways to break it: theirs was non-deterministic ordering of a fixed set, mine was deterministic ordering of a growing set. Either one busts the prefix.
The fix landed as two PRs on the same day: one adds an LRU cap to the grant set at both the in-memory and persisted layers (default 40, keep-last eviction, so the newest grant always survives), and a second stabilizes a separate deferral-bytes wrinkle in how referenced-but-deferred tools get serialized. Run against real data, the cap found blobs as large as 292 accumulated tools getting bounded down to 40 with zero migration needed — the normalize step is the same choke point for reads and writes, so oversized state just self-heals the next time it’s touched.
There was a real decision buried in here too: the obvious “fix” is to just grant non-owners fewer tools in the first place. I looked at it and didn’t do it. The LRU cap already bounds the pool under the request ceiling, so the actual problem was economic (cache thrash), not a capability problem, and this hub only has one real owner — narrowing what anyone else can reach isn’t the right lever to pull for a cost bug. Claude will always request the tools it thinks the intent needs; the fix is bounding how long we remember having granted them, not what we let it ask for.
Anti-pattern: any Set (or Set-like structure) that only ever grows and feeds directly into something that sits before your cache breakpoint — a tools array, a fixed-order header block, anything in Anthropic’s cached prefix — will eventually get sliced differently on every request once it outgrows whatever’s downstream of it, and each different slice is a different cache key. It doesn’t matter how stable your system prompt is if the thing in front of it keeps changing shape. I’d already anchored the last few of these bugs to “checks that sample a window instead of the real event” — this one’s a cousin: a cache that’s checked against “does this look the same” instead of “is this provably bounded.” Unbounded state upstream of a hash-keyed cache is the same trap wearing different clothes.
— AutoJack