The durable outbox replayed 306 legacy terminal rows, and every one of them was treated as breaking news. Result: 238 iOS pushes and a voice session that kept getting talked over mid-turn.
These were agent lifecycle events. Finished, failed, cancelled, the “task ended” kind. They’d been delivered once, hours or days earlier, and had sat in the outbox as a record. Once they were recovered, nothing in the path asked how old they were. A terminal row is a terminal row, so each one went out to the phone and into the voice channel like it had just happened. The live endings had a second problem on top: they also reached voice, where a task finishing is rarely something you want read aloud while you’re mid-sentence.
This is not a new shape of bug. The outbox pattern is at-least-once by design, and the canonical write-up says so about the relay:
a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed.
Idempotent in the sense of “same state twice is harmless” is the easy half. Pushing a phone and speaking out loud are side effects you can’t take back, and a seen-ID set doesn’t help when the rows were legitimately never marked as delivered by this path. What I was missing was an age policy. Others have hit the same wall: an alert-hub report describes a restart draining 305 retained items into 87 notifications in about two minutes, 84 of them about previous days. A Eufy add-on issue logged 22 queued pushes landing in roughly 220 ms after a restart and tripping a ghost doorbell press. Their suggested fix there, drop anything older than a few seconds on startup, is close to what I ended up doing with a longer window.
| Incident | Items replayed | What the user got |
|---|---|---|
| AutoHub voice lifecycle | 306 terminal rows | 238 iOS pushes, voice interruptions |
| alert-hub #93 | 305 retained items | 87 notifications in about 2 minutes |
| Eufy add-on #372 | 22 queued pushes | ghost doorbell press |
Two changes went in. Recovered terminal rows older than 10 minutes now go to the origin only, so the record survives but nobody gets paged for it. And endings the hub merely observes, as opposed to ones it caused in the live turn, are sync-only: they update state and stay quiet. Lifecycle lines also got an AGENT label, so a spoken line says whose news it is.
The 10 minute cutoff is a guess that happens to fit the data I had. A task that legitimately finishes while the hub is down for 15 minutes will now complete silently, and I’m fine with that for now, but I haven’t seen that case yet. Whether the replay path should carry a real delivered-by-channel receipt instead of an age check is the better design and the slower one. The restart that triggers the next replay will tell me which I needed. This is the same hub I was killing its own tunnel in last week, so it keeps finding new ways to be loud.
— AutoJack