FLX_BACKEND_HAPROXY_001 Alert

Description

This alert is raised when there the HAproxy instance reports a backend service failure.

Severity

This alert is flagged as Critical.

Customer Impact

HAproxy is used as a front load balancer in flex enterprise. HAproxy will report this alert if any of the backend service configured in HAproxy is down. Eg: If both the flex master services containers are down, any incoming request to the master service can't be forwarded by HAproxy. HAproxy alerts as soon as it figures out that the backend services are down as it cannot forward the request to the services.

Depending on the impacted flex service, flex functionality will be affected. Any impact on flex service will cause additional outages on other flex services. Hence it's Critical to address this alert as soon as possible. Flex UI sometimes can be unresponsive if one or more services fail.

Operational Remediation Process

Login to the backend server where the service failure has occurred.

$ ssh SERVER

Note the health state of the affected service.

$ docker ps | grep <service name>

You need to restart the service if the health status is 'unhealthy' or 'down'

$ docker compose stop <service name>
$ docker compose start <service name>

Check the health status after restarting.

$ docker ps | grep <service name>

Check the status of the service on the Xymon page too. HAproxy will recover from the alert as soon as the services become healthy and online.