Serial IO performance advantages
Hi all-
I'm working on a database project right now (http://jdbm.sourceforge.net/) that needs to have the absolute fastest mechanism for streaming data onto a file (this is for use in a database log file).
The log file is going to be implemented as a write only ring buffer (e.g. we will write byte 0 through MAX_FILE_SIZE, then start over again with byte 0).
In most native file systems, one can specify that a set of IO operations is going to be for purely sequential writing. This allows the OS to disable some types of read-ahead buffering, optimize paging operations for write only, etc... and provides the absolutely fastest write times.
To my knowledge, Java doesn't really provide the developer with an option to say that file access will be purely sequential and write only - but I'm wondering if maybe there are things we can do to implicitly force the jvm to make the correct low level system calls?
I suspect that using a standard FileOutputStream would probably be a sufficient indicator - but we need to overwrite existing data without completely truncating the file (FOS will either kill the file and start fresh or append to the existing file, neither of which is suitable for a ring buffer).
Another possibility is the FileChannel.write(ByteBuffer) method. Maybe the combination of using a FileOutputStream, then getting the channel and periodically resetting the position to 0 would force things to be optimal for write only ring-buffering?
Does anyone have any experience with the performance characteristics of Java 's IO classes in this kind of situation?
Thanks in advance,
- K
I'm quite familiar with NIO - what I'm asking about is whether anyone has experience with the specifics of managing the interaction between NIO (or the earlier Java I/O mechanisms) and the capabilities of the operating system for optimizing sequential write only output.
As an FYI, several of the aspects of my original post (FileChannel, ByteBuffer, etc...) are direct references to NIO. The question is how to use NIO to optimize disk access for this particular situation.
Anyone else?
Thanks!
- K
> To my knowledge, Java doesn't really provide the
> developer with an option to say that file access will
> be purely sequential and write only - but I'm
> wondering if maybe there are things we can do to
> implicitly force the jvm to make the correct low
> level system calls?
That's a dangerous path because what works for a certain JVM at a specific point in time, may not work in another context.
The best option, especially in an open source project where you don't control how it's going to be used, is to go by the book and use NIO according to specification, not according to observed behaviour in a special case.
> That's a dangerous path because what works for a
> certain JVM at a specific point in time, may not work
> in another context.
I definitely don't see it that way. If I can make our application have better performance under even some JREs then it's worth doing, even if the performance may be 'normal' under other JREs. There are huge chunks of NIO that are not at guaranteed to perform any performance bennefit unless the JRE has been optimized for them, but it's still worth doing...
Regardless, the question is not *whether* this is something to do, but rather if anyone has any experience with it. The lack of responses makes me think the answer is "no"...
- K
