ioDrive Status Alerts
There are several fio-status alerts and SNMP traps that indicate the status of your ioDrive device(s). Most statuses indicate normal function, however some indicate problems or potential problems with your device(s).
This article explains the alerts that require action. Some alerts require you to contact Fusion-io Support for further assistance.
Flashback Protection | Flashback Protection Failed | Wear-Out Warning | Read-Only Mode | Powerloss Protection | Thermal Write Governing | Temperature alarm
Flashback Protection
Description
Like many other memory devices, NAND flash eventually fails with use. Those failures can be either permanent or temporary. Flashback redundancy is designed to address those chips that experience permanent failures, and provides additional protection above and beyond ECC (Error Correction Code) for soft failures.
Flashback provides a real-time RAID-like redundancy at the chip-level, without sacrificing user capacity or performance for fault tolerance.
Examples
fio-status
fct4 Attached as 'fioe' (block device)
Fusion-io ioDIMM3 320GB, Product Number:FS1-001-321-CS SN:XXXXX
ioDIMM3, PN:00119401203, Mfr:004, Date:20090529
Alt PN:FS1-SS1-321-CS
Flashback substitution active: 3/16
...
SNMP Trap
Not Applicable
Required Action
The device will function normally in flashback mode. Continue to monitor the device.
 |
As a best practice, always backup your data on a regular basis. Flashback protection mode does NOT signal impending failure, but it is a good reminder that devices can fail, and that data is best protected with proper redundancy. |
Flashback Protection Failed
Description
Flashback protection was protecting your device, but enough NAND flash has now failed, and the device is no longer usable.
Examples
fio-status
fct0 FAILED as 'fioa' (block device)
ERROR: Flashback protection exhausted. Too many NAND chip failures.
Flashback active on: 2/5
Fusion-io ioDrive 320GB, Product Number:FS1-002-321-CS SN:XXXXX
...
SNMP Trap
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmFlashbackTrap |
Critical |
Trap |
1.3.6.1.4.1.30018.0.1003 |
Operator intervention |
Immediate |
Always on |
Required Action
ioDrive device may need to be replaced. Run fio-bugreport and send file to Fusion-io Support.
Wear-Out Warning
Description
NAND flash erase blocks will wear out with use. The ioMemory VSL will remap the device(s) and replace worn-out erase blocks with reserves as needed. When an ioDrive device's reserves are less than 10%, the software will issue a wear-out warning.
Examples
fio-status
fct21 Not attached
Fusion-io ioDrive 640GB, Product Number:FS1-004-640-CS SN:411502
ioDrive 640GB, PN:00214100808, Mfr:003, Date:20110412
...
Media status: WARNING: Nearing capacity wear-out; Reserves: 4.02%, warn at 10.00%
SNMP Trap
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmWearoutTrap |
Major |
Trap |
1.3.6.1.4.1.30018.0.1001 |
Operator intervention |
Day |
Always on |
Required Action
ioDrive device needs close monitoring as it approaches 0% reserves and goes into write-reduced mode, which will result in reduced write performance. Prepare to replace the device soon.
Read-Only Mode
Description
The remaining reserves are depleted. At 0% reserves, the device entered write-reduced mode. Soon after entering write-reduced mode, the device enters read-only mode. All attempts to write to the device will fail.
Examples
fio-status
fct4 Attached as 'fioe' (block device)
WARNING: READ-ONLY MODE. ALL WRITES WILL FAIL!
Internal error.
Fusion-io ioDIMM3 320GB, Product Number:FS1-001-321-CS SN:XXXXX
...
SNMP Trap
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmNonWritableTrap |
Critical |
Trap |
1.3.6.1.4.1.30018.0.1002 |
Operator intervention |
Immediate |
Always on |
Required Action
The ioDrive device may need immediate replacement. Run fio-bugreport and send file to Fusion-io Support for confirmation of diagnosis.
 |
Verify that the alert is not a false positive. An out of memory condition or over temperature condition may cause a drive to go into a read-only state. |
Powerloss Protection
Description
ioDrive device has capability for power loss protection but it is disabled.
All but the earliest versions ioDrive devices are equipped with powerloss protection. In the event of an unplanned power loss, the ioDrive will continue to write all of the committed writes using power from one or more on board capacitors.
Examples
fio-status
fct4 Attached as 'fioe' (block device)
WARNING: Powerloss protection available but DISABLED
Fusion-io ioDIMM3 320GB, Product Number:FS1-001-321-CS SN:XXXXX
...
SNMP Trap
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmPowerlossProtectTrap |
Major |
Trap |
1.3.6.1.4.1.30018.0.1007 |
Operator intervention |
Day |
Always on |
Required Action
Run fio-bugreport and send file to Fusion-io Support for troubleshooting. Powerloss protection should not be disabled, and is it not a common event.
Thermal Write Governing
Description
The ioDrive will start throttling write performance once the on board controller temperature reaches 78°C. If the controller temperature continues to rise, the ioDrive will shut down once the controller temperature reaches 85°C.
This warning appears once the temperature threshold of 78°C has been exceeded.
Examples
fio-status
fct1 Attached as 'fiob' (block device)
Fusion-io ioDrive Duo 1.28TB, Product Number:FS3-204-641-CS SN:XXXXX
ioDIMM3 640GB MLC, PN:00276700504, Mfr:003, Date:20110106
Located in slot 1 Lower of ioDrive Duo SN:91849
WARNING: Thermal write governing activated, performance may be
limited. If this condition persists, increase air flow,
lower room temperature or reduce write load.
Total write governing level: heavy
...
SNMP Trap
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmTempHighTrap |
Major |
Trap |
1.3.6.1.4.1.30018.0.1004 |
fusionIoDimmTempOkTrap |
Immediate |
always on |
Required Action
The ioDrive is not being cooled sufficiently. Increase fan speeds of the server, decrease the ambient temperature, reduce write load, or move the ioDrive device to a different slot.
If you are successful in decreasing the temperature, the fio-status alert and SNMP Trap will clear. Here is the trap that is sent as the trap clears:
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmTempOkTrap |
Informational |
Trap |
1.3.6.1.4.1.30018.0.1005 |
N/A |
Informational |
sent once |
Note temperature of the ioDrive. Drive temperature should continue to decrease.
Temperature Alarm
Description
Temperature has reached 85°C, and the ioDrive device has shut down.
Examples
fio-status
fct2 FAILED
Fusion-io ioDrive Duo 1.28TB, Product Number:FS3-202-641-CS SN:72057
ioDIMM3 640GB MLC, PN:00276700501, Mfr:004, Date:20101015
Located in 0 Upper slot of ioDrive Duo SN:42860
WARNING: Temperature alarm triggered.
Temperature is above 84.7 degrees C.
...
SNMP Trap
A fusionIoDimmNonWritableTrap will be sent as the ioDrive is shutdown and goes into a temporary read only mode.
| Trap Name |
Priority |
Trap/Poll |
OID |
Clear |
Time Frame |
Trap Config |
| fusionIoDimmNonWritableTrap |
Critical |
Trap |
1.3.6.1.4.1.30018.0.1002 |
Operator intervention |
Immediate |
Always on |
Required Action
Increase fan speeds of the server, decrease the ambient temperature, reduce write load, or move the ioDrive device to a different slot. Then reboot the system to re-enable the device.