Over-Subscribed Bus
Problem
It is possible to install multiple ioDrives and other PCI-Express peripherals in a way that causes unequal performance from each of the drives. In networking, this is frequently called over-subscription, and it is normal for most system designs. For PCI-Express bus based high-performance peripherals, having an over-subscribed topology can drastically affect performance, especially if the drives are RAIDed in some fashion.
The following figure illustrates a balanced bus vs. an over-subscribed bus. System A has 16 PCI lanes connected from ioDrives to a switch chip, and 16 lanes from the switch chip to the root complex. This results in a balanced topology. System B has 12 PCI lanes connected from ioDrives to a switch chip, but only 8 lanes from the switch chip to the root complex. This causes an over-subscription condition, decreasing performance.
Note: Keep in mind that other PCI-E devices such as high performance network cards and graphic cards can also create an over-subscription condition.
Figure 12 – Balanced bandwidth (A) vs. over-subscription (B)
Solution
To verify that there are no bandwidth bottlenecks in the PCI-E bus, it is important to run the fio-pci-check utility and look for errors.
Problem
A related common performance problem that is harder to diagnose is when the PCI Express lanes are run off of the south bridge. Running off the south bridge is inherently slower than running off the north bridge. This can create an oversubscribed configuration that fio-pci-check may not be able to diagnose.
Solution
Doing this diagnosis is beyond the scope of this version of the document. For assistance in determining if your system suffers from this problem, you need to collect a fio-bugreport and the results of the system-vetting benchmarks. Send the bundle to support@fusionio.com, requesting assistance in debugging a performance issue.
Below is an example of a bandwidth error reported by fio-pci-check, caused by an over-subscription condition.
Figure 13 – Bandwidth error, over-subscription