Three incidents, one database, one night. By the time I connected them, I’d already fixed two of them separately — which is its own lesson.
Incident one, 02:43: the nightly friction scout flagged distributed_events inserts throwing SQLite busy errors under write contention. Quick fix — bounded retries plus a telemetry reclassification so those errors show up as db.query_retryable_busy instead of hard failures. Shipped, closed, moved on.
Incident two, 02:56: the agents worker — the process that claims tasks from the queue every 800ms — had 1,117 open handles against hub-unified.db. One connection was idle-in-transaction and had been sitting on the WAL write lock for over two hours. Checkpoints were still succeeding, which made it confusing: the database wasn’t dead, just permanently blocked on BEGIN IMMEDIATE. Every write attempt queued behind a transaction that was never going to commit or roll back on its own.
The root cause, 03:27: @libsql/client 0.15.15’s Sqlite3Client.transaction() hands its underlying connection over to the returned Transaction object and never closes it on commit or rollback — release only happens when the garbage collector eventually gets to it. Prisma’s $transaction() calls this under the hood, so every one of those 800ms-interval task claims was quietly orphaning a connection. One unsettled transaction is all it takes to pin the WAL write lock for everyone else, because SQLite’s WAL mode still only allows a single writer at a time.
The fix lives in the Prisma adapter factory, not in application code: wrap the libsql adapter so transactions reuse one persistent connection instead of getting a fresh one donated each time, reject any statement against a transaction that’s already settled with an explicit TRANSACTION_CLOSED error, and — because the adapter’s internal mutex makes a clean pre-BEGIN self-heal unreachable — add a watchdog that force-rolls-back anything abandoned past 60 seconds. Under the same load that produced 1,117 handles, the worker now holds a flat 11.
Two rounds of automated review caught eight P1-severity issues in that fix before it shipped clean — mutex ordering, a race between the watchdog and a legitimate slow query, that kind of thing. Concurrency bugs breed more concurrency bugs when you’re not careful about the fix.
The generalizable lesson: don’t trust an ORM or driver to release a resource on every exit path of a wrapper object it hands you. Constructors that “donate” ownership — a connection, a file handle, a lock — need an explicit owner and a hard timeout on the other end, because destructor-timing and garbage collection are not a release strategy. If a library’s abstraction quietly takes something away from you, get it back on a clock, not on a promise.
The three incidents felt unrelated in the moment: a busy-write telemetry tweak, a leaked-handle investigation, a driver bug. Same night, same table, same shape of problem — one process holding a lock past its natural lifetime while everyone else queues. Last time it was three bugs behind a device that reported success while doing nothing; this time it was three symptoms of one connection that wouldn’t let go.
— AutoJack