FLX_BACKEND_JAVA_001 Alert

Description

This alert is raised when there is a high JVM heap usage on containers.

Severity

This alert is flagged as Critical.

Customer Impact

High memory usage can cause issues on the server , as Linux will have less memory for caches making it more inefficient

Flex memory usage depends on the number and usage of the different microservices so it can be a difficult issue to pinpoint.

Depending on the impacted server, multiple flex services might get impacted due to a shortage of memory.

Chances are there will out of memory occurring on containers and getting killed to insufficient memory.

Operational Remediation Process

Login to Xymon and see the server memory usage pattern.

  1. log in to the server in question and for which we got alert
  2. Execute command free -m to see the free memory and swap space
$ top -b -o +%MEM | head -n 22

You can use docker ps to translate from the container id to the service name

The yellow/memory alerts might be caused by the following issues:

  1. services nodes - arangodb-server /Rabbimq/Redis/Mongo leak memory
  • fix: restart the affected containers
  1. non-services nodes - flex service memory leak
  • fix: restart the affected flex service. If the issue persists even after restarting, it requires further investigation from the developer team for a memory leak.
  1. none of the above - the server might simply not have enough memory to run all the apps. Review the memory usage of other nodes and consider either of following

a) Moving applications around to different servers where memory is available

b) Deploying new instances and moving applications.

c) Increasing the size of instances and moving applications