FLX_BACKEND_ELASTIC_009 Alert
A critical Alert is raised when the incoming events in logstash_pipeline are not getting processed.
This alert indicates that the incoming logs are not being processed by Elasticsearch. Hence this may result in the failure of log capture and Kibana won't be showing any logs.
Login to the server which is hosting the logstash and elasticsearch services. Usually its the monitoring server in the environment.
$ ssh SERVER
Here you need to check the health status of two services , logstash and elasticsearch.
Note the health state of the affected service.
$ systemctl status logstash
$ systemctl status elasticsearch
Get the logs of the services.
$ cd /var/log/logstash
$ cat *
$ cd /var/log/elasticsearch
$ cat *
You can give it try to restart the logstash service and wait for it to become active. Chances are that after the logstash restart, the log events can be processed. You should also check the logs of Elasticsearch and restart elastic search services if there are errors mentioned in Elasticsearch service.
$ systemctl restart logstash
$ systemctl restart elasticsearch
Both services should be active and running to process the incoming logs.
The logs of the services will give more insight into the actual issue. We need to troubleshoot further with specified errors in the services.