irqbalance: Avoid overloading CPU-0 with interrupt requests

Introduction

To facilitate the high performance nature of the ioDrive in multi-card installations, it is important to distribute the system's interrupt requests across multiple CPU cores. This leads to the best possible performance and prevents the possibility of CPU soft lockups. You can accomplish this by Installing and running irqbalance on your system.

Symptoms

In a multi-card installation, poor performance may be due to one or more of the following:

• CPU soft lockups
• 2009-11-16 09:43:17 [2919351.551998] BUG: soft lockup - CPU#0 stuck for 61s! [fct3-worker:6679]
• /proc/interrupts shows all ioDrive interrupts being directed to CPU0, even in high I/O situations
• /proc/interrupts shows ioDrives using XT-PIC interrupts

Checklist

1. Make sure that the kernel boot parameter noapic is not being used. The output from the following grep command should be empty.

$ grep noapic /proc/cmdline


2. Make sure irqbalance is installed.

$ rpm -qa irqbalance
irqbalance-0.55-15.el5


3. Check the interrupt class, which may be one of MSI, IO-APIC, XT-PIC. Depending on your driver/firmware version, different types of interrupts are recommended. With driver series 1.2 IO-APIC is typically preferred, and with driver series 2.0 MSI are preferred.

$ grep iodrive /proc/interrupts
177: 81939 6288307 0 0 IO-APIC-level ehci_hcd:usb1, uhci_hcd:usb5, iodrive
185: 81287 184139 4518006 0 IO-APIC-level uhci_hcd:usb3, uhci_hcd:usb8, iodrive

Setup

$ yum install irqbalance

More Information

The following examples show how running irqbalance decreases the amount of interrupts on CPU0.

Running without irqbalance

$ cat /proc/interrupts
 CPU0 CPU1 CPU2 CPU3 
. . .

177: 6835411 0 0 0 IO-APIC-level ehci_hcd:usb1, uhci_hcd:usb5, iodrive
185: 5625768 0 0 0 IO-APIC-level uhci_hcd:usb3, uhci_hcd:usb8, iodrive

Running with irqbalance

$ cat /proc/interrupts
 CPU0 CPU1 CPU2 CPU3 
. . .

177: 81939 6907062 0 0 IO-APIC-level ehci_hcd:usb1, uhci_hcd:usb5, iodrive
185: 81287 184139 4975320 0 IO-APIC-level uhci_hcd:usb3, uhci_hcd:usb8, iodrive

For documentation, see http://www.irqbalance.org/documentation.php