Out of Memory Error - using Http download
Hi,
I am trying to download a file from a remote location and below is the code I am using. It throws a java.lang.OutOfMemoryError
for large files, say around 150 MB.
Can you please advise how to fix it to read the large files.
Thanks!
URL url =new URL("http","<IP_ADDRESS>",8080,"<FILE_PATH_NAME>");
con = (HttpURLConnection)url.openConnection();
is = con.getInputStream();
bufIn =new BufferedInputStream(is);
fout=new FileOutputStream("C:\\Clips\\longclip.3gp");
baos =new ByteArrayOutputStream();
System.out.println("Downloading File...");
int c = bufIn.read();
while(c != -1)
{
baos.write(c);
c = bufIn.read();
}
baos.writeTo(fout);
[1142 byte] By [
singalga] at [2007-11-26 16:37:34]
# 1
> Can you please advise how to fix it to read the large files.Don't read them into memory. Write them directly to your local file.
ejpa at 2007-7-8 23:04:13 >
# 2
Thanks ejp.
Really, thats what I would luv to do.
Can you please suggest how to make that possible?
If I change the while loop to:
while(c != -1)
{
baos.write(c);
c = bufIn.read();
baos.writeTo(fout);
}
Is it what you mean?
Thanks very much for your help. You are a gem.
Also, I think you have some books on Java networking/sockets....are you a writer? If yes, can you please provide me the link for the books (to buy or to download)?
Message was edited by:
singalg
Message was edited by:
singalg
# 3
Well, I tried the above and tested it. The problem is that the while loop never seems to end. I think it goes in infinite loop.Can you please advise a fix.Message was edited by: singalg
# 4
See my books page at http://www.telekinesis.com.au/wipv3_6/Books.A21.Forget your second loop, it is rubbish. Use your first code but don't use a ByteArrayOutputStream, use a FileOutputStream wrapped in a BufferedOutputStream.
ejpa at 2007-7-8 23:04:13 >
# 5
Thanks for your advise and for the URL.
Below is the modified code which is used for testing and here are the findings:
1. It took about 10.5 minutes to download the file of size (approx) 150 MB. Is there a faster way to do so? (Like download using multiple threads?)
2. The size of original file is: 154,026,650 bytes where as the size of downloaded file is: 154,026,496 bytes. Do you know what could cause this difference?
Below is the modified code:
long startTime = System.currentTimeMillis();
URL url = new URL("http","10.176.96.64",8080,"/prod/tst/un/tst/prod/longclip.3gp");
con = (HttpURLConnection)url.openConnection();
is = con.getInputStream();
bufIn = new BufferedInputStream(is);
fout=new FileOutputStream("Z:\\Clips\\cache\\longclip.3gp");
BufferedOutputStream bos = new BufferedOutputStream(fout);
System.out.println("Downloading File...");
int c = bufIn.read();
while(c != -1)
{
bos.write(c);
c = bufIn.read();
}
System.out.println("Download Complete!");
long stopTime = System.currentTimeMillis();
System.out.println("Total Time Taken to download: "+(startTime-stopTime));
# 6
You need to flush or close the BufferedOutputStream.
You could get rid of the buffered streams and read/write to/from a large byte array:
byte[] buffer = new byte[64*1024]; // play with this quantity to explore performance
int count;
while ((count = in.read(buffer)) > 0)
out.write(buffer, 0, count);
out.close();
ejpa at 2007-7-8 23:04:13 >
# 7
Thanks ejp.
I tried both the approaches: using byte array and using bufferedoutputstream. Here are the results as per my experience:
1. BufferedOutputStream works faster than using byte array.
2. If using BufferedOutputStream, No need to specify any buffer size.
Hence, I will be using BufferedOutputStream for my implementation.
Thanks very much for your help.
