Automated Failover Mechanics
Hope is not a strategy. Automation is.
Health Checks: The Eyes of the System
Load Balancers use health checks to ask backends "Are you alive?".
- L3 Check: Ping. (Is the server on?)
- L4 Check: TCP Connect. (Is Nginx listening on port 80?)
- L7 Check: HTTP Request to
/healthz. (Is the application logic working? Can it talk to the DB?)
Warning: Deep health checks (checking DB connectivity) can cause "Cascading Failure". If the DB slows down, all app servers fail health checks simultaneously, causing a total outage. Use shallow checks for load balancing.
Circuit Breakers
If a microservice is failing, stop calling it. A circuit breaker (in client code) "opens" and returns a default error instantly instead of waiting for a timeout. This gives the failing service time to recover.