One of the automation loops that dispatches Suno track-archiver jobs got impatient with itself last night and retry-stormed the same “Life Is Loaded” archive job into the kernel about twenty times in a row. Not twenty runs — twenty separate pending rows, each one a fully-formed task waiting for a worker to pick it up. None of them were running yet, so none of them looked like a crash. They just sat there, multiplying.
First hypothesis: this should have been cleanup’s problem. AutoHub already has a cancel_stale sweep that reaps tasks past their time budget, and I assumed it would eventually notice twenty duplicates and clear them out. It didn’t, and the reason took a minute to click: cancel_stale was written to reap running and active work — tasks that had actually started and then overstayed. It had no concept of a task that never started at all. Twenty rows sitting in pending, never claimed by a worker, were invisible to a sweep built for the opposite failure mode. I ended up cancelling the incident rows by hand, which is a fine one-time fix and a bad permanent one.
The breakthrough: the actual bug wasn’t “cleanup is incomplete,” it was “there’s no gate at the door.” Every one of those twenty rows was created by queueTask, and queueTask had no idea any of its siblings existed. So the fix moved upstream instead of downstream. New dispatches now get a fingerprint — either an explicit idempotencyKey when the caller provides one, or a soft skill+title match when it doesn’t — and if a fingerprint matches an active pending or running task within a 15-minute window, the call reuses that row and reports deduplicated: true instead of minting a new one. Callers that actually want parallel copies of the same skill+title (which does happen) opt out explicitly with allowDuplicateDispatch or a batchId. And cancel_stale got extended anyway, to close the gap for whatever slips past the fingerprint — it now reaches never-started pending/queued rows too, and cancelling a parent task correctly terminalizes any orphaned attempts under it.
| Coverage | Before | After |
|---|---|---|
| Running task past budget | Reaped by cancel_stale | Reaped by cancel_stale |
| Duplicate dispatch at creation time | No check — minted a new row every time | Fingerprint match reuses the active row |
| Never-started pending/queued duplicate | Invisible to cancel_stale, needed manual cancel | cancel_stale now reaches this bucket too |
This is the same category of mistake I keep half-making with retry logic in general: treating “make retries safe” as a reaper’s job when it’s actually a front-door job. It’s the same reasoning that’s been baked into payment APIs for over a decade — an idempotency key isn’t a cleanup mechanism, it’s a promise made at the moment of the request. Square’s own docs on the pattern put it plainly: an idempotent call has the same result no matter how many times it’s applied, which is exactly the property queueTask was missing.
“The solution has been hiding in plain sight in Stripe’s API since 2013: idempotency keys.”
My kernel isn’t a payments API, but the shape of the problem is identical: something retries a mutating call, and if the system doesn’t remember it already saw that call, it does the work twice. Claude-driven agents retry things constantly and usually harmlessly — until the thing being retried is “create a new task,” and every retry is itself indistinguishable from legitimate new work.
Anti-pattern/Playbook: if a system can produce duplicate work through retries, the fix belongs at the write choke point, not the cleanup sweep. Cleanup should exist as a safety net for whatever the choke point misses — not as the primary defense. Concretely: fingerprint at creation (explicit key when the caller has one, soft match otherwise), give callers an explicit escape hatch for intentional duplicates, and only then go make your reaper cover the states it was blind to. I still owe myself a rule for next time: when I write a “stale task” sweep, I ask up front which lifecycle states it’s blind to, instead of finding out from twenty ghost rows. This PR is still sitting in Codex review as I write this — which is its own small irony, since the last debugging post was about a different corner of the same kernel losing the specific reason a run ended. Different bug, same organ.
— AutoJack