Memory Overload!!!
I create a Vector of size 1.000.000 (which of cource increases dramatically during execution) and i get a memory error mesage
Exception in thread"main" java.lang.OutOfMemoryError: Java heap space
at COM.ibm.db2.jdbc.app.DB2ResultSet.SQLGetStringData(Native Method)
at COM.ibm.db2.jdbc.app.DB2ResultSet.getString2(Unknown Source)
at COM.ibm.db2.jdbc.app.DB2ResultSet.getString(Unknown Source)
what can i do about that?
the command that adds data to my vector is
temp = i_ut.concat(" , ").concat(i_de).concat(" , ").concat(r_ut).concat(" , ").concat(r_de);
results.add(temp);
where i_ut , i_de , r_ut , r_de are stings
and results is my Vector
[810 byte] By [
georousa] at [2007-11-26 17:08:09]
> what can i do about that?
Stop trying to load up EVERYTHING into memory at once. Looks like you are querying a database; I bet you can keep that data in the database and just query in "chunks" depending on what you are doing with the data. I bet you don't NEED to have it all in memory at the same time.
The thing is that i need to work with all the queried data.Do you think it is a good idea to store them in to a file?
> The thing is that i need to work with all the queried data.
> Do you think it is a good idea to store them in to a file?
Lessee, if you need to work with large amounts of data, you might need something to store it in. A "data store." No, that doesn't sound quite right ... the data is going to be the base of everything you do ... oh, I got it! A "data base".
In other words, keep your data in the database where it already lives, and write SQL queries that summarize it or select just the pieces that you need at the moment.
You won't be able to do anything more efficiently if all the data is in memory, and you certainly won't be able to if it's stored in a file that you have to maintain.
> The thing is that i need to work with all the queried
> data.
I'm sure that's true. But it's almost never true that you need to have access to all that data at the same moment in time. If you'll be more specific, I'm certain that there is a less memory-intensive way to do whatever you're trying to do.
Also, using concat will create new Strings that you don't need. You might want a StringBuilder or StringBuffer.
So captain obvious advises me to store this query in my database and then manip the data. but isn't it obvious that the data will always have the same size no matter where they are stored?i'm going to see whether string builder uses more politely my memory.Thnx!
> So captain obvious advises me to store this query in
> my database and then manip the data. but isn't it
> obvious that the data will always have the same size
> no matter where they are stored?
Sure it will use the same size, but i'm willing to bet you have more hard drive space (where a database is stored) versus the amount of ram you have.
> i'm going to see whether string builder uses more
> politely my memory.
Um, you're still building a 1,000,000 (or more) entry collection in memory, and I bet this depends on how much data you are querying. That's still a problem, because even if that many happen to fit in memory, what if the data grows to 1,000,000,000 for example? Then what?
Your application design is not designed for scalability which is very important.
> So captain obvious advises me to store this query in
> my database and then manip the data.
What are you doing with 1 million rows of data?
Are you displaying them to the user? No user has the time or interest to look at 1 million rows of data. If you're just retrieving them to display, follow another poster's advice and request a small group of rows at a time. Or, if you're displaying query results, let the user decide how many results he/she wants.
More likely -- at least in the database applications that I've worked on -- you're summarizing the data, or selecting data items with particular criteria. And for those cases, you're far better using SQL and letting the DBMS do the summarization.
I'm trying to get the queried data so i can perform some modifications and create new data. Cptn.Obvious you advise me to store my queries in the DBMS and then gather the data needed? i have to get some strings from a JOIN function and try do find non-eucledian distances between them.
I am a student at http://www.it.teithe.gr
and not a commercial programmer
Thnx for your help!!
I haven't seen the obvious answer which is, increase the JVM heap size. It probably won't help you though. Have a nice weekend! :-)
