java.lang.OutOfMemoryError (new thread)

Hello friends,

I need an urgent assistance with a project that am working on presently at the moment.

I am trying to read and write data in byte arrays from a file with the use of FileInputStream and then write it to another file with the use of FileOutputStream.

My Code works very fine when handling files lesser than 30MB. When i intend to do such with files above 30MB i get a java.lang.OutOfMemoryError exception and the program stops reading and writing.

In order to be able to avoid this error, i then modified the amount of byte that is being read through FileInputStream to 1MB at a time, so that it keeps reading and writing 1MB of data. But i still get the java.lang.OutOfMemoryError exception and the process stops.

I really need to have this project completed asap. Please help rectify the problem. Below is my code:

I anticipate your prompt response guys.

publicclass Encodeextends Thread

{

File f;

File dir;

privateint MAX_ARRAY_COPY=1048576;

int fileSize;

int startPoint=1;

int startReadPoint=1;

String newFile;

int execCount=0;

public Encode(File ft)

{

f=ft;

dir=new File("SecuredMedia/"+f.getName().substring(0,f.getName().lastIndexOf(".")));

dir.mkdirs();

newFile=f.getName();

newFile=dir+"/"+newFile;

}

public File getEncodedDirectory()

{

return dir;

}

publicvoid run()

{

encodeFile();

}

publicvoid encodeFile()

{

try

{

FileInputStream fis=new FileInputStream(f);

fileSize=fis.available();

execCount++;

System.out.println("Execution Count: "+execCount);

SecureFile.encodeBtn.setEnabled(false);

SecureFile.status2.setText("Encoding");

byte[] red=newbyte[MAX_ARRAY_COPY];

byte[] reversed=newbyte[MAX_ARRAY_COPY];

System.out.println("MAX_ARRAY_COPY : "+MAX_ARRAY_COPY);

System.out.println("red : "+red.length);

int b=fis.read(red,startReadPoint,MAX_ARRAY_COPY);

fis.close();

System.out.println("Data Read : "+b);

int rcount=0;

for(int y=MAX_ARRAY_COPY-1;y>-1;y--)

{

reversed[rcount]=red[y];

rcount++;

}

FileOutputStream fos=new FileOutputStream(newFile);

fos.write(reversed,startPoint,MAX_ARRAY_COPY);

fos.close();

int tempFileSize=new FileInputStream(newFile).available();

System.out.println("Written File Size : "+tempFileSize);

if(tempFileSize < fileSize)

{

if((fileSize - tempFileSize) < MAX_ARRAY_COPY)

{

MAX_ARRAY_COPY=fileSize - tempFileSize;

}

startReadPoint=tempFileSize;

startPoint=tempFileSize;

System.out.println("Finished Logical Execution: "+execCount+"\n");

encodeFile();

}

else

{

}

SecureFile.status2.setText("Completed");

SecureFile.encodeBtn.setEnabled(true);

}

catch(IOException e)

{

e.printStackTrace();

}

}

}

[5193 byte] By [joe_ogunkalua] at [2007-10-2 8:41:59]
# 1
Increase the heap size.java -Xmx...
CeciNEstPasUnProgrammeura at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 2

In encodeFile() you call encodeFile() recursively. This allocates new space for the new method call local variables without releasing the previous call variables because the method has not finished.

I am not quite sure what you are trying to do here, your code looks a bit complicated for what it seems to be intended for, also relying on available() is not a good idea, what if you were reading from a socket or some other stream that cannot tell you its exact size. There are better ways to find our whether a stream is exhausted.

HTH

Mike

bellyrippera at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 3

thank you guys for the professional response, especially bellyripper making me understand the scope of the problem.

What i am trying to do here is to read the contents of a file, reverse it and then write it to another file. Basically what i want to do is reading from a file and writing to another file. Most of the file i will be reading and writing to another file will be large in size (i.e over 30 MB up to 2GB).

So when i try to read the whole file at once and then write it to another file, i get out of memory error. So i decided to read 1MB out of the file and then write it to the output file and so on till the whole file is read and written. That is my main motive. But in the cause, i wrote a code which has a logical error triggering a memory error.

So what is the best approach and the best code to use for this task.

joe_ogunkalua at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 4

>So what is the best approach and the best code to use for this task.

As usually - it depends. A possible approach is something like this:

open 'source' for input

open 'target' for output

while(not end of source file){

read a block from source //beware, at the end there will not be a whole block

write block to target

}

close files.

It's you who will have to write the code and decide what methods to write.

Mike

bellyrippera at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 5

> > open 'source' for input

> open 'target' for output

> while(not end of source file){

> read a block from source //beware, at the end there

> re will not be a whole block

>write block to target

> }

> close files.

>

>

> It's you who will have to write the code and decide

> what methods to write.

>

> Mike

thanks for the response. Don't you think it going to generated another OutOfMemoryError. And also , can i use this method to read up to 800MB of data.

Once again thanks for your assistance and support.

joe_ogunkalua at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 6
based on your experience, which one do you think will be the best alternative between the two:FileInputStream and FileOutputStreamRandomAccessFilewhich one do you think is of best performance and better to use in this case.
joe_ogunkalua at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 7
>> What i am trying to do here is to read the contents> of a file, reverse it and then write it to another> file.Why do you need to reverse the file?
sabre150a at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...
# 8

> > open 'source' for input

> open 'target' for output

> while(not end of source file){

> read a block from source //beware, at the end there

> re will not be a whole block

>write block to target

> }

> close files.

>

>

thanks very much mike for the logical structure you gave to me. I successfully read and wrote 83.1MB of data without getting any OutOfMemoryError.

Thanks once again for the help.

Joe....

joe_ogunkalua at 2007-7-16 22:44:08 > top of Java-index,Java Essentials,Java Programming...