Large data amount management
Hi
I'm currently writing a Bittorrent Client (like Azureus, but simpler...) within the scope of a University project.
It is already up and runs quite well. The real problem by now is that it is very RAM consuming and I know what the problem is, which is quite a luck...
The program currently downloads piece of files in a random order from remote peers. At the end, it writes all these pieces into one or several files, according to a predefined order. The problem is that now, the program keeps all downloaded pieces into memory (into some objects).
For small files, this is not a pain while being not very efficient. But as soon as it deals with large files, the problem occurs:
OutOfMemoryException: java heap space
I know this is due to the JVM to run out of memory and that there is way to provide more memory to it, but this is not the way it should be resolved, since the program should be able to download huge files ( tens of Gigas) while the memory not being that much...
So my idea is to periodically write the data in a file (or maybe another thing, like a buffer), which is not held into physical memory. But I must be able to write data in a correct order, so maybe at the beginning of the file, in-between or at the end also...
Could anyone help me find the best way to do this and save this **** physical memory?
Thanks a lot...
Bat
