FLX_BACKEND_ELASTIC_007 Alert

Description

A critical Alert is raised when the logstash heap space gets utilized over the limit.

Severity

This alert is flagged as Warning.

Customer Impact

This alert indicates that the logstash service is using the headspace over its threshold for the last 5 min.

During high heap space usage, the processing of logs will get slow and chances are that the logstash service will get failed, resulting in no logs.

Operational Remediation Process

Login to the server which is hosting the logstash services. Usually it's the monitoring server in the environment.

$ ssh SERVER

Note the health state of the affected service.

$ systemctl status logstash

Get the logs of the services.

$ cd /var/log/logstash
$ cat *

You can give it try to restart the logstash service and wait for it to become active. Chances are that after the logstash restart, the log events can be processed.

$ systemctl restart logstash

Note: Sometime logstash will take time to restart.

If restarting didn't help, try to increase the heap space manually on the logstash service temporarily and see if that helps to bring back the logstash service.

To temporarily increase the maximum size of total heap space. Open the logstash configuration file.

$ vim /etc/logstash/jvm.options

Change the value of Xms #Xmx represents the maximum size of total heap space -Xms1g to -Xms2g