transferring large files
Hi,
I have to transfer a file from client to the server, its working fine as long as the file size is small.
but when the file size increases it is giving me this error.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
so what exactly shud i do to transfer files of huge size.
how do i break it down and into small chunks and send it to the server
what is the modification required in the following code:
byte[] buf = new byte[1024];
BufferedInputStream bin = null;
ByteArrayOutputStream bout = null;
try {
FileInputStream fis = new FileInputStream( zipFile );
bout = new ByteArrayOutputStream();
byte[] b = new byte[1024];
int len = 0;
while( (len = fis.read(b, 0, 1024) ) != -1 ) {
bout.write( b, 0, len );
}
System.out.println("Fis available? 0 if yes : 1 if No "+fis.available());
bout.flush();
fileArray = bout.toByteArray();
[980 byte] By [
javitea] at [2007-10-2 21:28:51]
Well, if you load the whole file into memory, of course you're going to run out of memory at some point.
Are you doing RMI or just straight sockets? For sockets, get rid of the ByteArrayOutputStream and write directly to the socket.
For RMI, you will need to arrange to send reasonable-sized chunks at a time and have them reassembled at the other end. You can do this either via some kind of multi-stage API or by sending an object like the following:
public class SerializableFileContents implements Serializable
{
public static final long serialVersionUID = ...;
private File file;
public SerializableFileContents(File file)
{
this.file = file;
}
private void writeObject(java.io.ObjectOutputStream out)
throws IOException
{
out.defaultWriteObject();
FileInputStream in = new FileInputStream(file);
byte[] buffer = new byte[8192];
int count;
while ((count = in.read(buffer)) > 0)
{
out.writeInt(count);
out.write(buffer,0,count);
}
out.writeInt(count);
in.close();
}
private void readObject(java.io.ObjectInputStream in)
throws IOException, ClassNotFoundException
{
in.defaultReadObject();
FileOutputStream out = new FileOutputStream(file);
int count;
while ((count = in.readInt()) > 0)
{
byte[] buffer = new byte[count];
in.readFully(buffer);
out.write(buffer,0,count);
}
out.close();
}
}
You have to arrange for 'file' to mean the same thing at client and server: this is left as an exercise for the reader ...
I've arranged the internal protocol so that you can change the buffer size at the sending end without having to redistribute the .class file to the receiving end. I've also deferred all error handling to the moment when writeObject is called; you may wish to modify this so that the object can't be constructed if the file doesn't exist.
(Just to prove that I do occasionally write code ...)
Message was edited by:
ejp
ejpa at 2007-7-14 0:41:53 >
Quick question: Does RMI wait for the object serialization to complete before sending over the network or does RMI send data as serialization is happening based on the buffer size of the connection protocol?Message was edited by: Fergus
Each argument/result is transferred directly (e.g. not as a MarshalledObject).
ejpa at 2007-7-14 0:41:53 >
Apologies to the OP for hijacking the thread.By directly: Do you mean written to the underlying socket used for the connection?
Yep
ejpa at 2007-7-14 0:41:53 >
Hello ejp,
I'm working on a RMI peer-to-peer and I have difficulties to transfer the files. I had got the similary problem than javite (java.lang.OutOfMemoryError: Java heap) with the huge files.
So I try to use your SerializableFileContents class but it does not work (or I don't understand how it works... ;-)
How the "private" methods writeObject and readObject are called ?
How must I use the SerializableFileContents class with the ObjectOutputStream and ObjectInputStream ?
Can you give me an example for use ?
Thank you very much for your answer.
Thierry
Just use ObjectOutputStream.writeObject(). If you look into Serialization you'll find that it calls the private methods if they exist automatically.
ejpa at 2007-7-14 0:41:53 >
I'm also a starter in java. have been looking for a way to send files across a network via RMI also. I found this code only so far, so I was wondering if there was an example on how to use it also.
In my case I only need to send a file from the client to a preset destination on the server. I supposed there is a way to tell the server that the file has been send so the server can do with it whatever he needs to do with it ?
thanks in advance
well if you just have to transfer a file then here are the steps,
you have a remote interface, wich has a method say uploadFile which takes in a byte array.
In your client you have to read the desired file and call the remote method passing in the byte array.
the server program will implement the remote interface. that is it wil actually take in this byte array and write it to the destination you specify or do whate ever you want to.
thanks for the reply, however actually I meant using the above codesnippet. I do understand the basic steps, I just never worked with serialized objects.
So say I have a file to upload to the server: I guess I would have to do:
SerializableFileContents sfc = new SerializableFileContents(aFile);
the command to start reading that file on the serverside would be:
otdbRMI.loadComponentFile(aFile) // Call to native code viaRMI and JNI
Now what I was wondering was the stuff inbetween to get it to work (or a working example..)
Thanks
You need to add a File getFile() method.
You just call your remote method from the client, passing the object as a parameter. When your method executes at the server the content has already been received into a file called 'file'. (This means that it's important to ensure that 'file' has a relative not an absolute pathname, as it's going to be interpreted at the server as well as the client.) Then you just rename it or move it to where you want it.
You can improve this. When I wrote this about 8 years ago I had a technique for changing 'file' to a temporary filename when it is being read, which must have been something like 'this.file = File.createTempFile()' inside readObject() after the defaultReadObject() call. I don't remember the details but you can work it out I'm sure. This way you don't have to constrain the File to be operable on both platforms.
ejpa at 2007-7-14 0:41:53 >
