Scenario: The 502 Bad Gateway Spike
At 14:10 UTC, alerts fire across the edge layer: API Gateway is returning HTTP 502 Bad Gateway on 40% of inbound traffic. The backend application servers report low CPU (12%) and low memory (2.1GB / 16GB).
The candidate is provided with actual shell terminal snapshots captured during the incident.
# Snapshot 1: Application Health Check
$ curl -I [http://127.0.0.1:8080/health](http://127.0.0.1:8080/health)
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 15
# Snapshot 2: Kernel Network Buffer Metrics
$ cat /proc/net/sockstat
sockets: used 68240
TCP: inuse 120 orphan 4 tw 65400 alloc 68100 mem 42
RAW: inuse 0
FRAG: inuse 0 memory 0
# Snapshot 3: Socket States Breakdown
$ ss -s
Total: 68420
TCP: 68120 (estab 84, closed 65420, orphaned 4, timewait 65400)
# Snapshot 4: Application System Logs
$ tail -n 5 /var/log/app/backend.log
2026-09-25 14:10:02 ERROR [client.py:44] Cannot assign requested address
2026-09-25 14:10:02 ERROR [client.py:44] Cannot assign requested address
2026-09-25 14:10:03 ERROR [client.py:44] Cannot assign requested address
Q:Examine Snapshot 2 and 3. There are over 65,000 sockets in TIME_WAIT (tw 65400). What causes a socket to enter the TIME_WAIT state, and which side of the connection enters it?
Reveal â–¾
The endpoint that performs the Active Close (initiating the TCP connection teardown by sending the first FIN packet) enters the TIME_WAIT state after completing the four-way handshake.
The socket remains in TIME_WAIT for $2 \times \text{MSL}$ (Maximum Segment Lifetime, typically 60 seconds) to ensure two things:
- Any delayed or duplicate packets lingering on the Internet expire without corrupting a future connection reusing the same 4-tuple.
- The final
ACKsent to the remote peer is acknowledged; if dropped, the remote peer can retransmit itsFINsafely.
Q:Look at Snapshot 4: Cannot assign requested address (Linux errno 99: EADDRNOTAVAIL). Connect this error directly to the TIME_WAIT count. Why is the backend failing to make outbound calls?
Reveal â–¾
The server has suffered from Ephemeral Port Exhaustion.
By default, Linux allocates a range of local ports (defined in /proc/sys/net/ipv4/ip_local_port_range, typically 32768 to 60999, yielding ~28,232 ports) for outbound client connections to external services.
Because the application is initiating short-lived outbound HTTP calls (e.g., querying a database or auth service) and actively closing them, each connection consumes an ephemeral port and locks it in TIME_WAIT for 60 seconds. When the number of closed connections exceeds the available port range within a 60-second window, the kernel runs out of source ports for the destination IP, returning EADDRNOTAVAIL.
Q:The candidate recommends immediately setting sysctl net.ipv4.tcp_tw_recycle = 1. Why is this considered dangerous in modern networking, and what is the proper kernel parameter to reuse sockets safely?
Reveal â–¾
tcp_tw_recycle is extremely dangerous and was removed from the Linux kernel in version 4.12. It caches the per-host timestamp of the last packet. If multiple users or services communicate with the machine from behind a single NAT gateway (sharing an IP address), their out-of-sync internal clocks cause the kernel to silently drop valid connection attempts from other clients behind that NAT.
The safe sysctl setting is:
sysctl -w net.ipv4.tcp_tw_reuse=1
tcp_tw_reuse allows the kernel to safely reallocate an incoming socket in TIME_WAIT state for an outbound connection if the timestamp is strictly greater than the most recent timestamp received on the previous connection, complying with RFC 1323.
Q:Tuning kernel parameters is only a band-aid. What is the fundamental root cause in the application’s software architecture that led to this catastrophe, and how must the code be refactored? Reveal â–¾
The root cause is Connection Churn caused by the lack of HTTP Keep-Alive and connection pooling.
The application code is creating a brand-new TCP socket for every single outbound HTTP sub-request and closing it immediately upon receiving the response.
The permanent fix is to refactor the HTTP client to use a shared, persistent connection pool (keep-alive). Reusing existing TCP connections across requests completely eliminates ephemeral socket creation and bypasses the TIME_WAIT lifecycle entirely.
Variations & Real-World Impact
- Microservice Meshes: In Kubernetes setups with sidecar proxies (like Envoy / Istio), high traffic volume between local containers can trigger ephemeral port exhaustion over the loopback interface (
127.0.0.1) if the proxy does not pool upstream connections. - SO_REUSEADDR vs SO_REUSEPORT:
SO_REUSEADDRallows a server to bind to a local port currently inTIME_WAITstate (useful when restarting a server).SO_REUSEPORTallows multiple independent server processes to bind to the exact same IP and port simultaneously, letting the kernel balance incoming traffic across them.
Discussion & Comments