Programming using Direct I/O
See the article General Tuning Techniques for the motivation to use Direct I/O.
Linux
Under Linux, the best way to enable direct I/O is on a per-file basis. This is done by using the O_DIRECT flag to the open() system call. For example, in an applicatoin written in C, you may see a line similar to this:
fd = open(filename, O_WRONLY);
To make this file be accessed through unbuffered or direct I/O, you would change the line to this:
fd = open(filename, O_WRONLY | O_DIRECT );
Note that the ioDrive requires that all I/O performed on a device using O_DIRECT must be 512-byte alligned and a multiple of 512 bytes in size. Buffers used to read data to and write data from must be page-size aligned. For additional details, see the man pages for open(), getpagesize(), and posix_memalign().
The following is a simple application that writes a pattern to the entire ioDrive in 1MB chunks, using O_DIRECT:
#define _XOPEN_SOURCE 600
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#define __USE_GNU
#include <fcntl.h>
#include <string.h>
#define FILENAME "/dev/fiob"
int main( int argc, char **argv)
{
void *buf;
int ret = 0;
int ps = getpagesize();
unsigned long long int bytes_written = 0;
int fd;
if( (ret = posix_memalign(&buf, ps, ps*256)) ) {
perror("Memalign failed");
exit(ret);
}
memset(buf, 0xaa, ps*256);
if( (fd = open(FILENAME, O_WRONLY | O_DIRECT) ) < 0 ) {
perror("Open failed");
exit(ret);
}
bytes_written = 0;
while( (ret = pwrite(fd, buf, ps*256, bytes_written)) == ps*256) {
bytes_written += ret;
}
printf("Wrote %lld GB\n", bytes_written/1000/1000/1000);
close(fd);
free(buf);
}
Figure 26 – Writing a pattern in 1MB chunks
Note: In most cases, performance can be slightly enhanced by using pwrite instead of write.
Windows
DirectIO for Windows is set up through the CreateFile() call, using the FILE_FLAG_NO_BUFFERING flag (see http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx or search for “CreateFile” on http://msdn.microsoft.com/).
When doing Direct IO it is important to keep read/writes and buffers used for read/write sector size aligned. For a discussion of alignment requirements, see the “File Buffering” topic on MSDN at http://msdn.microsoft.com/en-us/library/cc644950(VS.85).aspx. The Fusion-io ioDrive's sector size defaults to 512 bytes.
C++ Code Sample
The following code sample writes to the block device using O_DIRECT. It has a class that encapsulates the operating system file functions that support O_DIRECT.
#include <unistd.h> #include <iostream> #include <errno.h>
#include <fcntl.h>
using namespace std; // Short demonstration of a container class that writes
using O_DIRECT.
// Compiled using GNU C++.
#define FILENAME "/dev/fioc"
class DirectFile
{
public:
DirectFile() {};
~DirectFile() { ::close(fd); };
int open(char* filename);
int write(char* buf, int size_t);
int close();
size_t gbytesWritten() { return bytes_written / 1000 / 1000 / 1000;
}
private:
int fd;
size_t bytes_written;
};
int DirectFile::open(char* filename)
{
int ret = 0;
fd = ::open(filename, O_DIRECT | O_WRONLY, S_IRUSR | S_IWUSR | S_IRGRP);
if( fd == -1 )
{
ret = -1;
}
bytes_written = 0;
return ret;
}
int DirectFile::write(char* buf, int bufsize)
{
int written = pwrite(fd, buf, bufsize, bytes_written);
if( written >= 0 )
bytes_written += written;
return written;
}
int main( int argc, char **argv)
{
char* buf;
int ret = 0;
enum
{
numpages = 256,
};
int pagesize = getpagesize();
int bufsize = pagesize * numpages;
if( ret = posix_memalign((void**)&buf, pagesize, bufsize) )
{
cerr << "Memalign failed" << endl;
exit(ret);
}
memset(buf, 0xaa, bufsize);
DirectFile file;
if( file.open(FILENAME) )
{
cerr << "Open of " << FILENAME << " failed" << endl;
exit(ret);
}
do
{
ret = file.write(buf, bufsize);
if( ret < 0 )
{
cerr << endl << "Error writing bytes to " << FILENAME << ". written=" << file.gbytesWritten() << "GBytes";
cerr << ", errno=" << errno << " " << strerror(errno) << endl;
}
else
{
cout << ".";
}
} while( ret > 0 );
if( ret >= 0 )
cout << endl << "Wrote " << file.gbytesWritten() << "GBytes" << endl;
free(buf);
return ret;
}