"Java heap space" Problem
I am developing a Program for which I need to use an integer array of int [9000][9000] and in future I may require even bigger to keep track of each and every thing.For the time I was using just one dimensional i.e. int [9000] and i was not putting other variable in loop.When I got that I have well developped the program i.e. it was giving good results so I thought to use two dimensional array as was my need.But it gives
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
error on defining two dimensioan array.How can I solve this memory problem?
[591 byte] By [
Kimsa] at [2007-11-27 1:53:57]
you can't store so much data with the default memory allocation of the JVM
you can try to load the jvm with
java -Xms=512M to start the JVM with 512megs of memory that can be allocated
anyway i'm not sure that it's a good idea to have so much data in your program ; it could be a better idea to use a database
Thanks sir I will try this but infact its not my requirement.It is someone who is giving me .txt file so I need to read this file.However thanks
Kimsa at 2007-7-12 1:24:53 >
Are you sure you need to have all that information in memory for the program to work?I would look for a way to process information in a "streaming" fashion, where you keep only as much information in memory as needed while reading the file.
Im having the exact sam problem at the moment.
My program is encoding a 23.6mb file.
Im reading in every byte in this file and outputting a result to another file.
Is this error due to poor programming?
Can the number of Loops/variables within the program influence this?
Thanks
Do not hijack another poster's thread.
In this thread, people already suggested you can increase
the memory allocated to Java by using -Xms (or, better yet, -Xmx).
If that DOES NOT work for your program, please post a NEW topic.
>
> Is this error due to poor programming?
>
Very likely. But without seeing the relevant part of your code, we can only guess.
>
> Can the number of Loops/variables within the program influence this?
>
Possible. But without seeing the relevant part of your code, we can only guess.
And again: do not hijack another poster's thread.
> Do not hijack another poster's thread.
>
> In this thread, people already suggested you can
> increase
> the memory allocated to Java by using -Xms (or,
> better yet, -Xmx).
> If that DOES NOT work for your program, please post a
> NEW topic.
>
> >
> > Is this error due to poor programming?
> >
>
> Very likely. But without seeing the relevant
> part of your code, we can only guess.
>
> >
> > Can the number of Loops/variables within the
> program influence this?
> >
>
> Possible. But without seeing the relevant part
> of your code, we can only guess.
>
> And again: do not hijack another poster's thread.
So am i supposed to create a new thread.
The reason i didnt was because people would reply with read the thread below!!!!
To Kims:
Besides all the good advice given in this thread so far, you might also
consider the use of sparse matrix. If you have a large array (say 9000x9000),
but most of the entries will be zero, then you can consider using a sparse matrix instead.
There are many free sparse matrix implementations on the internet,
each offering a different trade-off regarding space usage,
time per iteration, time per addition, time per deletion, etc. etc.
To carrics3:
> So am i supposed to create a new thread.
> The reason i didnt was because people would reply with read the thread below!!!!
You should have started a NEW thread to begin with.
Your problem (as I see them on your multiple OTHER threads) have nothing to do with a large 9000x9000 integer array.
I have tried java xmx=512M and for XMS also on Command promt in Bin Directory of JDK but It says unroginized option and Could not create JVMPleaseguide me
Kimsa at 2007-7-12 1:24:54 >
> I have tried java xmx=512M and for XMS also on
> Command promt in Bin Directory of JDK but It says
> unroginized option and Could not create JVM
Please type "exactly" what we said.
java -Xmx=512m YourClassName
rather than
java xmx=512M YourClassName
Addendum: More importantly, please read the other replies in this thread, and answer these questions:
1) Do you need all those data in memory AT THE SAME TIME?
If not, you should consider some sort of batch processing approach
2) Are the entries all non-zero? Or are many of them zero?
If so, you should consider using a sparse matrix.
Etc.
Adding 512m, 1024m, 2048m... is only a temporary fix.
Because later, when you have larger data, you will once again run out of memory.
Yes The matrix is already a sparse Matrix.I just need an Array of TOTAL size i.e. 9000 that I have created so I go on manipulating it.
Kimsa at 2007-7-12 1:24:54 >
Let me copy this function fo you.OUT2 is the Global Array that I am manipulating for each variable.This represents the distance of a variable from other vertex.For example if out2 [1234]=3 for variable (vertex) 1, its mean vertext 1234 is at a distance of 3 from vertex 1 and then i m writing it in a file also in form of a matrix like 1 1234 3 .I need array because I need previous distances also for next processings.
public void Sh_Fun(int source)
{
//////////////////////////////////
String out1=" ",rec;
//Glob.rec_counter++;
boolean found=false;
////////////////////////// Preparing List of Functions that are passed to this recursive function /////////////
if (Glob.RV.indexOf(Integer.toString(source))==-1)
Glob.RV=Glob.RV.concat(Integer.toString(source).concat(" "));
//////////////////////////////////////////////////////////////////////////////////////////////////////////
try
{
//////////////////// open and close files //////////////////////////
// open the input file
RandomAccessFile dis = new RandomAccessFile( "ListeSortant.txt", "rw" );
///////////////////////////////////////////////////////////////////////////////////////////////
while ( (rec=dis.readLine() ) != null )
{
String[] s1=rec.split("\\s"); // one line read
if (source==Integer.parseInt(s1[0]))
{
out1=out1.concat(s1[1].concat(" "));
found=true;
if (source==Glob.x)
Glob.direct=out1.concat(s1[1].concat(" ")); // JUST to create a string of vertices that are directlt attached with source
}
else
{
if (found==true)
break;// because a particular pattern of source exits in .txt file consectively
}// end of else
}// end of while reading file
////////////////////////////////////////////////////////////////////////////////////////////////////////////
out1=out1.trim(); // to eliminate the leading and trailing spaces
String[] s2=out1.split("\\s");// now s2 contains all the vertices attached directly with source (s2 and out1 contain the same thing but s2 contains no spaces
//////////////////////////////////////////////////////////////////////////////////////////////////
if (out1.equals("")) //infact when no vertex is attached directly with a passed source then problem occurs
return;
///////////////////////////////////////////////////////////////////////////////////////////////////////////
// System.out.println("Source Passed to This function is ="+source+" length of s2 is ="+s2.length);
// for (int k=0;k<s2.length;k++)
//System.out.println(" Total Vertices Attached to Vertex "+source+ " are= "+s2.length+" and attached vertices are="+s2[k]);
////////////////////////////////////////////////// MAIN LOOP of Writing in OUT2 /////////////////////////////////////////////////////////////////////
Glob.ps.println("This is for vertex "+source);
for (int i=0;i<s2.length;i++)
{
if (source==Glob.x)
{
Glob.out2[Integer.parseInt(s2[i])]=1;
Glob.ps.println(Glob.x+ " "+s2[i]+" 1"); // writing to output file
} // end of if
else
{
if (Glob.RV.indexOf(s2[i])==-1 && Glob.out2[Integer.parseInt(s2[i])]==0 && Glob.direct.indexOf(s2[i])==-1)//The condition Glob.direct.indexOf(s2[i]==-1 is for checking that if the source is othe than Glob.x then it should not change the distance of vertices directlt aattached to Glob.x, it should change only others
{
Glob.out2[Integer.parseInt(s2[i])]=Glob.out2[source]+1;
Glob.ps.println(Glob.x+ " "+s2[i]+" "+(Glob.out2[source]+1)); // writing to output file
}
} // end of else
}// end of for loop for i
Glob.ps.flush();
/////////////////////////////////////////////////// Recursive CALL /////////////////////////////////
for (int j=0;j<s2.length;j++)
{
if (Glob.RV.indexOf(s2[j])==-1)
Sh_Fun(Integer.parseInt(s2[j])); // Recursive Call
}
//////////////////////////////////////////////////////////////////////////////////////////////////////
dis.close();// Close our input stream
}// end of try
// Catches any error conditions
catch (IOException e)
{
System.err.println ("Unable to read from file");
System.exit(-1);
}// end of catch
}// end of function
Message was edited by:
Kims>
Kimsa at 2007-7-12 1:24:54 >
> Yes The matrix is already a sparse Matrix.I just
> need an Array of TOTAL size i.e. 9000 that I have
> created so I go on manipulating it.
o... kay... is there a question in there? Are you still having trouble?
or does "java -Xmx=1024m YourClassName" solve your problem?
>
> Let me copy this function fo you
>
o... kay... what about it? Does it have syntax error? other error? what?
Please Read previous reply also with code so that you can understand.So it executes but after 15 minutes,it through an exception of STACKOverFlow Error
Exception in thread "main" java.lang.StackOverflowError
at sun.nio.cs.SingleByteEncoder.encodeArrayLoop(SingleByteEncoder.java:91)
at sun.nio.cs.SingleByteEncoder.encodeLoop(SingleByteEncoder.java:130)
at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:252)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:111)
at java.io.PrintStream.write(PrintStream.java:476)
at java.io.PrintStream.print(PrintStream.java:619)
at java.io.PrintStream.println(PrintStream.java:756)
at Sh_D.Sh_Fun(Sh_D.java:171)
at Sh_D.Sh_Fun(Sh_D.java:206)
at Sh_D.Sh_Fun(Sh_D.java:206)
at Sh_D.Sh_Fun(Sh_D.java:206)
at Sh_D.Sh_Fun(Sh_D.java:206)
206 is recursive call
Message was edited by:
Kims
Kimsa at 2007-7-12 1:24:54 >
it GiVES error "invalid maximum heap size"I write java -Xms=1024m Sh_D(also tried for 512)
Kimsa at 2007-7-21 20:15:37 >
> I write java -Xms=1024m Sh_DWrite -Xms1024minstead.
Lokoa at 2007-7-21 20:15:37 >
cross post http://forum.java.sun.com/thread.jspa?threadID=5164490
Stack overflow is a completely different kind of problem from out of memory. It means you've written your recursive method wrongly, so that it always calls itself and you get infinite recursion.
For recursion to work there always has to be an end condition where the method does not call itself, and every recursive descent has to reach that condition at some point.
Changing heap size limit (-Xmx1024m by the way - no "=") certainly won't help - even increasing the maximum stack size will probably not help.
Yes Sir, I Think This is problem.I will check for this problem nw.However I Thank you all.You are all great ppl here.I will check for it and I hope I will solve it.Thanks once again
Kimsa at 2007-7-21 20:15:38 >
malcolmmc is right, you need to look at your recursion.
1) Sorry I mistyped the syntax for heap. It's "-Xmx512m", not "-Xmx=512m".
2) Your are NOT using a sparse matrix! Your "out2" is still a regular array.
I know that because you're using "[ ]" to access its elements.
A sparse matrix is an object that uses clever tricks to reduce/compress the array behind the scene.
3) But malcolmmc's point is even more important:
your recursion algorithm appears to be buggy.
In particular, EVERYTIME you enter Sh_Fun,
as long as s2.length>0, you might call Sh_Fun again. for (int j=0;j<s2.length;j++)
{
if (Glob.RV.indexOf(s2[j])==-1)
Sh_Fun(Integer.parseInt(s2[j])); // Recursive Call
}
>
Sir I have thought for infinite call of Recursion but I think it is correct.For the first variable going for call, it should execute for s2.length and then finish.WHat do u say about it sir.
Kimsa at 2007-7-21 20:15:38 >
> Sir I have thought for infinite call of Recursion but
> I think it is correct.For the first variable going
> for call, it should execute for s2.length and then
> finish.WHat do u say about it sir.
The first call will then call itself for s2.length times, that's correct.
But... in the subcalls, what is the value of s2.length? Is it zero? (I don't have time
to read your entire code to find out). But if s2.length is still > 0, then you will
have subsubcalls, and then subsubsubcalls, and then subsubsubsubcalls...
It is for you to find out. I'm only volunteering time here.
Addendum: Now, it's time for you to go back to your code,
and do some System.out.println() or run it through a debugger.
Because StackOverflow almost always always always means
you have an infinite loop. So, go away, fix it, then come back if you have
specific questions.
Sorry first of all but I never meant that you are being payed for this job.Its ur greatness as I already said.
However Sorry.
Let vertex 1 has 10 childs (then each child has 5 child and so on ) but ultimately and eventually,when this function has been executed for one, it is quite clear from condition that when j=s2.length() ,its finished.For zero,I have put a condition that when it is zero,it should return
That is y I was saying that May be there is some other problem.
again sorry Sir If you dont have time, what you have done is enough for me.
Message was edited by:
Kims
Kimsa at 2007-7-21 20:15:38 >
Oh, I'm not offended. I was just telling you that it is your turn now to do some investigation.
> Let vertex 1 has 10 childs (then each child has 5
> child and so on ) but ultimately and eventually,when
> this function has been executed for one, it is quite
> clear from condition that when j=s2.length() ,its
> finished.
Your "reasoning" sounds good. But your java execution says "infinite loop" (with 99.999% probability).
So go back and re-examine them.
(In particular, you say you have a set of vertices and each one connects
to other vertices. Are there cycles? Could you go back to the same vertice
over and over again, and thus reach an infinite loop?)
Cycles can occur in this case but I have put condition over this also as you can see in recursion call, I have condition if (Glob.RV.indexOf(s2[j])==-1)
RV contains all the vertices already passed to this function so it will pass only those vertices that have not been passed already.
Of course I am thinking to revise again
Kimsa at 2007-7-21 20:15:38 >
> Cycles can occur in this case but I have put
> condition over this also as you can see in recursion
> call, I have condition if
> (Glob.RV.indexOf(s2[j])==-1)
>
> RV contains all the vertices already passed to this
> function so it will pass only those vertices that
> have not been passed already.
I know your "reasons" sound good. But your code shows there's a problem.
Stop trying to convince "me". I don't care whether your code works or not. You care.
>
> Of course I am thinking to revise again
>
Don't think. Use a debugger or use some println() to show the execution trace.
You can think and look at your code for 10 years
and not notice a small typo (perhaps "<" instead of "<=", or "==" instead of "!=")
Use a debugger or something. Good luck. Hope you solve your problem soon.
