Database admin pilot fish is on call and sleeping soundly when his beeper goes off at 1 a.m.

But it’s not an alert from monitoring software or from the trouble center. The phone number on the pager is one fish doesn’t recognize.

He calls the number. It’s a user who tells him that a job on the server is stuck, and he’s not getting new rows into a table.

Bleary-eyed fish logs on and checks. Sure enough, the job has a problem. A single mouse click restarts the job, and it’s working again.

But fish also notices in the log that the job has been stuck since 10:30 a.m. — two days before.

“Yeah, it’s been annoying,” user explains when fish asks about it. “The stuck job has been paging me to tell me there’s a problem every 15 minutes.”

To read this article in full, please click here