The 2 A.M. Outage: Expert Core Command Line Diagnostics for High-Stakes Incidents

The pager on my desk screamed at 2:17 AM, the harsh red light of the PagerDuty alert cutting through the dim glow of my laptop screen. I’d been asleep for 47 minutes when the first notification popped up: *Payment Service Error Rate > 40% for 90 seconds*. By the time I blinked the sleep out of my eyes, Slack’s #incident-response channel had already exploded with 137 unread messages: the CTO was @-ing everyone on call, the customer success team was reporting thousands of failed payment attempts from merchants, and the monitoring dashboard showed our core PostgreSQL transaction database was rejecting 62% of incoming connection requests. A quick back-of-the-envelope calculation told me we were losing ~$19,000 per minute in unrecoverable transaction revenue—if we didn’t fix this in 13 minutes, we’d blow past the $2.3M loss threshold the board had flagged as a “material adverse event” for the quarter.

I’d spent the last three years building the incident response playbook for NovaPay, the 120-person fintech startup I led infrastructure for, and this was the first real test of the “core CLI first” diagnostic workflow I’d written. No fancy APM tools, no cloud provider console—just raw, unfiltered Linux command line utilities, the kind that have existed since the first Unix kernels, but almost no one uses to their full potential. My goal was simple: isolate the failure domain in 5 minutes, fix it in 7, and get back to bed before my 6 AM standup.

First 60 Seconds: Triage and Scope

The first rule of incident response is *stop the bleeding before you diagnose the wound*. I didn’t open a log viewer or SSH straight to the database server. First, I ran a one-liner to confirm the blast radius: ```bash curl -s http://payment-api.internal/health | jq '.components.database.status' ``` The response came back in 12ms: `"status": "outage", "error": "connection pool exhausted, timeout waiting for idle connection after 10s"`. That was my first clue: this wasn’t a database crash, a disk failure, or a network partition. It was a connection pool saturation issue. But I still needed to confirm it at the system level, because application logs lie—sometimes connection pool errors are a symptom of a deeper problem, like a slow query hogging all connections, or a disk full of WAL files blocking new writes.

Next, I ran the 10-second diagnostic script I’d pre-built for exactly this scenario, saved to `/usr/local/bin/fast-diag.sh` on every production server: ```bash #!/bin/bash

fast-diag.sh: Core system metrics for incident triage

echo "=== 1. System Load ===" uptime echo -e "\n=== 2. CPU Top 5 Consumers ===" ps -eo pid,user,%cpu,%mem,stat,cmd --sort=-%cpu | head -6 echo -e "\n=== 3. Memory Top 5 Consumers ===" ps -eo pid,user,%cpu,%mem,stat,cmd --sort=-%mem | head -6 echo -e "\n=== 4. Disk Usage ===" df -hT --output=source,size,used,avail,pcent,target echo -e "\n=== 5. Inode Usage ===" df -i --output=source,itotal,iused,iavail,ipcent,target echo -e "\n=== 6. Network Connection Summary ===" ss -s echo -e "\n=== 7. PostgreSQL (Port 5432) Established Connections ===" ss -tnp state established '( dport = :5432 )' | wc -l echo -e "\n=== 8. Zombie