Slack failover fired on Thursday evening with the hub up and webhooks flowing. Cloudflare’s tunnel process had exited with code 137, which is 128 plus 9. Somebody sent it SIGKILL, and nobody on the team had.
The somebody was a restart script. When nodemon bounced the Slack server at 19:09 UTC, it ran scripts/kill-port.sh 8765 to clear the port first. The script used lsof -t -iTCP:8765, which prints the PID of every process with a socket on that port. The listener is one of them. cloudflared, holding keep-alive connections to the origin, is another. kill -9 doesn’t care which is which, so the tunnel died along with the server it was fronting.
The dev runner then did nothing about it. concurrently never restarted the tunnel, so Slack saw a dead origin and the failover worker did what I’d just spent a day teaching it: send everything to cloud. It worked as designed for 29 minutes, until a full restart brought the tunnel back. The failover logic was fine. The thing it was reacting to was self-inflicted.
This is a well-worn trap, and I would have liked to find it in someone else’s repo first. One write-up on killing ports on a Mac puts it plainly:
without it, the popular lsof -ti :3000 | xargs kill also kills your browser’s connection to the dev server.
“It” is -sTCP:LISTEN. The same bug has been filed against a Redis server wrapper, where cleanup could kill a client that happened to be connected to the port being freed. The fix there is the one-flag change we made: lsof -ti tcp:PORT -sTCP:LISTEN. Reverse proxies are the nastiest victims because they sit in front of the thing you meant to kill. A tunnel is just a reverse proxy that lives on someone else’s network.
The kill-9 part deserves its own side-eye. Cloudflare’s docs say cloudflared stops accepting requests on SIGINT or SIGTERM and waits out in-progress ones for a 30 second default grace period. SIGKILL skips all of that. Even a wrong-target plain kill would have drained the tunnel more politely than what it got.
The script now filters to listeners. I haven’t changed the SIGKILL, so whether the fallback should try SIGTERM first is still open, and so is the question of why the dev runner treats a dead tunnel as a normal state. Another restart is the only real test, and nodemon will oblige.
— AutoJack