FLX_BACKEND_ELASTIC_001 Alert

Description

A critical Alert is raised when there is a High JVM heap usage on the backend elastic-search service.

Severity

This alert is flagged as Critical.

Customer Impact

This alert indicates that the High JVM heap usage on the backend elastic search service has exceeded 95%.

A high JVM usage will cause the elastic search cluster to fail.

The usual cause for high JVM heap usage happens when there is an ongoing reindex in place or flex index services are trying to index too many assets that need to be processed by Elasticsearch. Elasticsearch service is crucial for search functionality in flex.

Any outage on elasticsearch will affect the search in flex.

Operational Remediation Process

Log in to the servers which are showing the high JVM heap usage on Elasticsearch.

In the flex backend, the Elasticsearch service is configured to run on 3 node cluster.

$ ssh SERVER

Get the elastic search container name.

$ docker ps | grep <service name>

Get the memory usage stats of elastic search.

$ docker stats <container_name>

This will show the overall usage of elasticsearch memory. You can get the logs of the elasticsearch service using

$ docker logs <service name>

Before you could restart any elastic search service, you need to check the health state of the other two elastic search services running on other service node. See if the other elastic search are healthy.

If the JVM heap usage is not decreasing, it's recommended to stop the elasticsearch service, increase the mem_limit and Heap_space of the elasticsearch service in docker-compose.yml. Then start the elasticsearch service again with the new heap space configuration.

You can stop the elasticsearch service.

$ docker compose stop <container name>

Edit the docker-compose.yml file to update the mem_limit and HEAPSPACE to a higher value.

$ cd /flex
$ vim docker-compose.yml

search for the service paragraph and change the Mem_limit to a bigger value, e.g: mem_limit: 2G increase it to mem_limit: 4G.

Similarly change the HEAPSPACE to bigger value to match it up with mem_limit, e.g: HEAPSPACE=1g increase it to HEAPSPACE=2g.

After changing the memory limit and heap space, it's required to restart the container.

$ cd /flex
$ docker compose up -d <service name>

Repeat the same steps on the remaining two elasticsearch services if a JVM heap space issue occurs on those hosts.