Basic ioDrive Troubleshooting on Linux
Basic ioDrive Troubleshooting on Linux
NOTE: The following assumes that the ioDrive utility ("fio-util" / "iodrive-util"
rpm) software is installed. For more information on this, please see
the ioDrive User Guide for Linux.
Follow the instructions based on the
version of your driver.
(jump to: Troubleshooting for Drivers 1.2.x)
Troubleshooting for Drivers 2.1.x
Please verify that the ioDrive driver is loading correctly.
$ fio-status
Found 1 ioDrive in this system
fct0 Attached as 'fioa' (block device)
Fusion-io ioDIMM3 160GB, Product Number:FS1-001-161-ES SN:6168
Firmware v42895 'Released'
161 GBytes block device size, 198 GBytes physical device size capacity.
PCI:0c:00.0, Slot Number:6
Internal temperature: avg 42.3 degC, max 44.3 degC
Media status: Healthy. 99.44% blocks good.
(The ioDrive red LED stays on until the iodrive driver is
successfully loaded. The description of all valid LED color
combinations is found in the ioDrive User Guide for Linux or
the Knowledgebase article entitled "Understanding
the ioDrive External LEDs".)
If the fio-status command is not successful, then run:
$ fio-pci-check
Root Bridge PCIe 6000 MB/sec
Bridge 00:06.00 (0c-0c)
Needed 1000 MB/sec Avail 1000 MB/sec
ioDrive 0c:00.0 Firmware 42895
This should display each of the ioDrives in the system. If this does
not report any ioDrives present, then the following steps can be useful
for more low-level troubleshooting.
The following steps can help you troubleshoot basic ioDrive problems
on Linux:
-
Verify that the ioDrive is seated properly in the server, that
it is in a PCI express (PCIe) x4 or greater slot for maximum
performance, and that the ioDrive is seen on the PCIe bus:
$ lspci | egrep -i '1aed|ioDimm'
0c:00.0 Mass storage controller: Fusion-io ioDimm3 (rev 01)
This makes sure that the ioDrive is seen at the most basic operating
system level. This means that the ioDrive is visible on the PCIe bus.
There should be one line of output for every ioDrive present in the
system. An ioDrive Duo has two ioMemory modules per drive and thus has two lines in the lspci output.
-
Verify that the software is installed properly:
To verify the driver:
# Red Hat or CentOS
$ rpm -qa iomemory-vsl\*
# Debian or Ubuntu
$ dpkg -l iomemory-vsl\*
$ rpm -qa iomemory-vsl\*
iomemory-vsl-2.6.18-92.el5-2.1.0.240-1.0
To verify the utilities:
# Red Hat or CentOS
$ rpm -qa fio\*
# Debian or Ubuntu
$ dpkg -l fio\*
$ rpm -qa fio\*
fio-firmware-42895-1.0.noarch
fio-util-2.1.0.240-1.0.x86_64
These examples above show the 2.1.0 driver installed along with the
corresponding firmware and utilities.
- Verify that there
are not conflicting versions of the drivers and utilities installed on
the same system. There should only be one driver installed. If another
driver version
had been present, there would be additional "iomemory-vsl . . . "
lines listed.
Older versions of the driver: Also
verify that there are no previous versions of the driver or utilities
installed (the name of these packages begins with "iodrive").
# Red Hat or CentOS
$ rpm -qa iodrive\*
# Debian or Ubuntu
$ dpkg -l iodrive\*
$ rpm -qa iodrive\*
iodrive-driver-1.2.4-1.7_2.6.18_92.el5
iodrive-firmware-1.2.4-1
iodrive-util-1.2.4-1.7
This example shows the 1.2.4 driver installed along with the
corresponding firmware and utilities.
-
If there are conflicting versions of the drivers and/ or
utilities, uninstall them by running these commands:
# To uninstall the driver
$ rpm –e <driver name>
# To uninstall the utilities
$ rpm –e <utility name>
- Check to see whether the ioDrive firmware is out of date and
needs to be upgraded. If the upgrade is needed, fio-status will report
"Firmware is out of date. Update firmware." There will also be a line
in /var/log/messages to tell if a firmware upgrade is required. The
firmware version can also be checked with the fio-pci-check utility
which is part of the Fusion-io utilities package ("fio-util"
rpm). The fio-pci-check utility is demonstrated at the beginning of
this article. ioManager can also report on the status and version of
the firmware.
-
Make sure that the ioDrive driver is permitted to load at boot
time. In /etc/modprobe.d/iomemory-vsl.conf there is a line that should be
commented out with a "#" at the start of it, as shown here:
$ cat /etc/modprobe.d/iomemory-vsl.conf
# To keep ioDrive from auto loading at boot, uncomment below
# blacklist iomemory-vsl
-
Make sure that the ioDrive driver loads on your kernel. The
driver is built for a specific Linux kernel version, which is part of
its package name. If the running Linux kernel version does not match
the Linux kernel version for which the driver was built, the driver
will not be loaded. This can occur, for instance, when your
kernel is updated.*
$ rpm -qa iomemory-vsl\*
iomemory-vsl-2.6.18-92.el5-2.1.0.240-1.0
$ uname -r
2.6.18-92.el5
In this example, the driver was built for the 2.6.18-92.el5 kernel and
matches the running Linux kernel version.
If they do not match, download the ioMemory driver source package from the Fusion-io support web site
(
http://support.fusionio.com), and rebuild it for your Linux kernel.*
Please follow the rebuild instructions at the following links.
Red
Hat: Building the ioDrive Driver From Source (Red Hat, Oracle, CentOS)
Ubuntu/Debian:
Building the Driver From Source
*NOTE: The 2.1.x driver series takes advantage of the Red Hat (RHEL)
5.x Application Binary Interface (ABI) to help avoid the need for
rebuilding after a minor kernel update.
Troubleshooting for Drivers 1.2.x
Please verify that the ioDrive driver is loading correctly.
$ fio-status
Found 1 ioDrive in this system
fct0 Attached as 'fioa' (block device)
Fusion-io ioDIMM3 160GB, Product Number:FS1-001-161-ES SN:6168
Firmware v17350 'Released'
161 GBytes block device size, 198 GBytes physical device size capacity.
PCI:0c:00.0, Slot Number:6
Vendor:1aed, Device:1003, Sub vendor:1aed, Sub device:1010
Internal temperature: avg 42.3 degC, max 44.3 degC
Media status: Healthy. 99.44% blocks good.
(The ioDrive red LED stays on until the iodrive driver is
successfully loaded. The description of all valid LED color
combinations is found in the ioDrive User Guide for Linux or
the Knowledgebase article entitled "Understanding
the ioDrive External LEDs".)
If the fio-status command is not successful, then run:
$ fio-pci-check
Root Bridge PCIe 6000 MB/sec
Bridge 00:06.00 (0c-0c)
Needed 1000 MB/sec Avail 1000 MB/sec
ioDrive 0c:00.0 Firmware 17350
This should display each of the ioDrives in the system. If this does
not report any ioDrives present, then the following steps can be useful
for more low-level troubleshooting.
The following steps can help you troubleshoot basic ioDrive problems
on Linux:
-
Verify that the ioDrive is seated properly in the server, that
it is in a PCI express (PCIe) x4 or greater slot for maximum
performance, and that the ioDrive is seen on the PCIe bus:
$ lspci | egrep -i '1aed|ioDimm'
0c:00.0 Mass storage controller: Fusion-io ioDimm3 (rev 01)
This makes sure that the ioDrive is seen at the most basic operating
system level. This means that the ioDrive is visible on the PCIe bus.
There should be one line of output for every ioDrive present in the
system. An ioDrive Duo has two ioMemory modules per drive and thus has two lines in the lspci output.
-
Verify that the software is installed properly and that there
are not conflicting versions of the drivers and utilities installed on
the same system:
# Red Hat or CentOS
$ rpm -qa iodrive\*
# Debian or Ubuntu
$ dpkg -l iodrive\*
$ rpm -qa iodrive\*
iodrive-driver-1.2.4-1.7_2.6.18_92.el5
iodrive-firmware-1.2.4-1
iodrive-util-1.2.4-1.7
This example shows the 1.2.4 driver installed along with the
corresponding firmware and utilities. There should only be one
"iodrive-driver- . . . " line present. If another driver version had
been present, there would be additional "iodrive-driver- . . . " lines
listed.
-
Check to see whether the ioDrive firmware is out of date and
needs to be upgraded. If the upgrade is needed, fio-status will report
"Firmware is out of date. Update firmware." There will also be a line
in /var/log/messages to tell if a firmware upgrade is required. The
firmware version can also be checked with the fio-pci-check utility
which is part of the Fusion-io utilities package ("iodrive-util . . ."
rpm). The
fio-pci-check utility is demonstrated at the beginning of this section.
ioManager can also report on the status and version of the
firmware.
-
Make sure that the ioDrive driver is permitted to load at boot
time. In /etc/modprobe.d/iodrive there is a line that should be
commented out with a "#" at the start of it, as shown here:
$ cat /etc/modprobe.d/iodrive
# To keep ioDrive from auto loading at boot, uncomment below
# blacklist fio-driver
-
Make sure that the ioDrive driver loads on your kernel. The
driver is built for a specific Linux kernel version, which is part of
its package name. If the running Linux kernel version does not match
the Linux kernel version for which the driver was built, the driver
will not be loaded. This can occur, for instance, when your
kernel is updated.
$ rpm -qa iodrive-driver\*
iodrive-driver-1.2.4-1.7_2.6.18_92.el5
$ uname -r
2.6.18-92.el5
In this example, the driver was built for the 2.6.18-92.el5 kernel and
matches the running Linux kernel version.
If they do not match, then download a pre-built version of the driver from the Fusion-io support web site
(
http://support.fusionio.com). There is a limited set of
pre-built drivers on the support web site. If your kernel does
not match any of the pre-compiled binaries, you must build the driver
installation package.
Please follow the rebuild instructions at the following links.
Red
Hat: Building the ioDrive Driver From Source (Red Hat, Oracle, CentOS)
Ubuntu/Debian:
Building the Driver From Source