FLX_INFRA_OS_003 Alert

Description

A critical Alert is raised when IO waits for a single CPU core in a server to be high last 15 minutes.

Severity

This alert is flagged as Critical.

Customer Impact

This alert indicates that there is a sudden high iowait of each core instead of the average of the CPU cores.

High iowait may mean that the hard disk or network is busy.

High I/O wait means the CPU is outstanding on requests, but further investigation is needed to confirm the source and effect.

High I/O wait forces the server to handle higher bandwidth to perform other computations while waiting on IO requests. I/O wait is a common metric to analyze system performance. When high I/O wait occurs frequently, it can negatively impact system performance by slowing down the system or causing issues such as low transaction throughput in the database and health degradation in applications and databases.

Operational Remediation Process

We need to identify the process or container which is causing the high IOwait

The two commands most commonly used to identify and troubleshoot I/O wait time are top and vmstat

the top command gives the average usage of Iowait. vmstat displays the statistics since the system’s last reboot. It can also show real-time statistics that update after a specified interval.

Once you’ve confirmed that you’re dealing with a high I/O wait time, the next step should be finding processes responsible for these I/O operations. One of the easiest and most frequently used commands for this is iotop.

$ pidstat -dl 20
$ iotop

One other important factor to consider is what disks are being used in these operations. The iostat command provides this information.

Eg: To display disk I/O statistics every 2 seconds for 5 times, the following command can be used:

$ iostat -xd 2 5

If I need to check the IOPS for a specific device, such as /dev/sda, I will run the command like this. Here the interval is 3s.

$ iostat -xd 3 /dev/sda

Fixing high I/O wait time

  1. Upgrade the hard disk and RAM in the server if you see a history of IOwait usage.

  2. Make sure the swap is turned off

  3. Check for any potential memory leak—some applications can consume memory without freeing it up after use. This can lead to a memory leak and eventually to a high I/O wait time.