General Tuning Techniques

Using Direct, Unbuffered, or Zero Copy I/O

Traditional I/O paths include the page cache, a DRAM cache of data stored on the disk. (See http://en.wikipedia.org/wiki/Page_cache for further details on page caching.) The ioDrive is fast enough that this and other traditional optimizations, such as I/O merging and reordering, and page caching, are actually detrimental to performance. I/O merging and reordering are eliminated naturally by the ioDrive, but the page cache must be bypassed at the application level.

Direct I/O bypasses the page cache. This allows the same memory regions written by the application to be DMA-transferred (http://en.wikipedia.org/wiki/Direct_memory_access) or copied directly to the ioDrive, rather than having to be copied into kernel-owned memory first, thus eliminating a copy. (See also http://en.wikipedia.org/wiki/Virtual_memory for additional interesting reading.)

Bypassing the page cache provides the following benefits:

• Less complex write path
• Lower overall CPU utilization
• Less memory bandwidth usage

In most cases, direct I/O is a big win for ioDrive-based systems, but you should benchmark your app to be sure of this. Some apps that don't use AIO, threads, or multiple processes to create multiple outstanding requests may benefit from the page cache instead.

Many I/O-intensive applications have tunable parameters that control how they interact with the low-level IO subsystem, including turning on direct I/O. Application tuning is outside of the scope of this document but will be covered in application- specific notes published by Fusion-io.

For other applications, it is necessary for the application provider to enable direct I/O or to modify the source to enable direct I/O and then recompile.

For a more in-depth look at how to write C code utilizing Direct IO, see Programming for Direct IO in Linux, using O_DIRECT.

dd Support

More recent versions of dd support the oflag=direct and iflag=direct parameters. These enable direct I/O for either the file being written to or the file being read from, respectively. Use oflag=direct when writing to an ioDrive and iflag=direct when reading from an ioDrive.

ioZone Benchmark

ioZone supports the -I (capital "eye") option to enable Direct I/O.

fio Benchmark

Fio uses the setting direct=1 it the job file, or --direct=1 on the command line to enable Direct I/O.