Umapping memory mapped buffers
I'm currently mapping a file contents to memory using ByteBuffer.map method and no issues there. The issue is that I can't unmap buffers, nor can I predict when the buffers are unmapped. My test machine is a Win XP SP2, AMD2100, 2Gig-Mem with 7200RPM disk.
When I close the underlying FileChannel or RandomAccess file, the memory is still not released from the mapped buffers. I verify this using an external process monitor. It is only released when I run the System.gc() manually.
So my question is, what is the best strategy for memory management in Java when memory mapping huge files. My test file is only 400Megs in size, but other files my be upto 500Gigs. I do mapping in various large buffer sizes of about 10Meg in size. I'm considering just running the GC manullay before every buffer mapping since they are in huge chunks, but I don't like this approach. My worry here is that I can not wait for the regular GC to kick in since I'm allocating huge amounts of memory outside the Java VM scope.
BTW: my code is achieving increadible performance of 1.6M variable length records/second read and parsed using this model.
The overall algorithm is simple: map large chuck of file into memory, create a wrapper object (a CapturePacket in my API), create a ByteBuffer view using slice() for the record header, parse the header within the view buffer, create a second view buffer using slice() for the records content (Packet data) and return the created CapturePacket object. And work on the next packet. My tests are able to do the 1.6M times a second, that pretty fast for Java.
