OutOfMemoryError - Help needed

Hi everyone,

i have the following function in which a query is executed several times, the results are stored in a Vector and the Vector then is stored in a class handling data.

publicvoid getAllSNPsPerGene(Gene g, GeneDataStorage store)

{

SNP s;

Vector<SNP> vec =new Vector<SNP>();

String schema;

if(_study.toLowerCase().equals("study1"))

schema ="study1.";

else

schema ="";

int chr = getChromosomeOfGene(g);

int tabs = getNumberOfSubtables(chr);

Statement stmt =null;

try

{

stmt = getStatement();

}

catch(SQLException e)

{

e.printStackTrace();

}

if(stmt ==null)

return;

ResultSet rs;

for(int j=1; j<=tabs; j++)

{

_query ="SELECT a.a_1, a.a_2, a.snp FROM "+schema;

_query +="my_table"+chr;

_query +="_"+_study+"_pt"+j+" a ";

_query +="INNER JOIN schema.map b using(snp) ";

_query +="WHERE not b.snp=null and b.gene='"+g.getId()+"' ";

_query +="and not a.a_1='-' order by b.snp_id";

System.out.println("Query "+_query+" num of tabs "+tabs);

try

{

rs = stmt.executeQuery(_query);

rs.beforeFirst();

while(rs.next())

{

s =new SNP();

s.setId(rs.getString("snp"));

s.setAlleles(rs.getString("a_1")+

rs.getString("a_2"));

vec.add(s);

s =null;

}

}

catch(SQLException e)

{

e.printStackTrace();

}

}

store.setSNPs(vec);

}

Now what i don't get is that it works very well for around 30 tables and then i get the error. After i process the data - usually the above method retrieves the data found in 15 tables, then the data are processed and then i enter this method again - i call the following method to delete the data and to free memory:

publicvoid clearSNPs()

{

SNP s =new SNP();

for(int i=0; i<_snps.size(); i++)

{

s = _snps.get(i);

s =null;

}

_snps.clear();

}

and before i set the Vector i call clear (just to be sure):

publicvoid setSNPs(Vector<SNP> snps){

if(_snps !=null)

_snps.clear();

_snps =null;

this._snps = snps;

}

So, again, the first methode retrieves the data that fit one criterion, the data are processed, deleted and then the methode is called again and so on. Another strange thing: The process dies when executing a query that does not retrieve a lot of data...

Right now i am really at a loss concerning how i could free enough space or better why the memory is not freed...

[4656 byte] By [abenstexa] at [2007-11-26 18:11:24]
# 1

1. Use a PreparedStatement instead of the mess of String concatenation that you have.

2. In your "rs.next()" loop, setting "s = null;" is useless. Delete that line.

3. This method is useless:

public void clearSNPs()

{

SNP s = new SNP(); // useless construction.

for(int i=0; i<_snps.size(); i++)

{

s = _snps.get(i);

s = null; // useless--you only set local reference to null--

// the reference to the SNP is still in _snps

}

_snps.clear();

}

All you need is:

public void clearSNPs(){

_snps.clear();

}

Assuming _snps contains the only references to the created SNPs, they will be eligible for garbage collection. You could instead create a new Vector:

public void clearSNPs(){

_snps = new Vector<SNP>();

}

4. This method is useless:

public void setSNPs(Vector<SNP> snps) {

if(_snps != null)

_snps.clear();

_snps = null;

this._snps = snps;

}

All you need is:

public void setSNPs(Vector<SNP> snps) {

this._snps = snps;

}

5. Use ArrayList instead of Vector. If you need synchronization, use Collections.synchronizedList.

doremifasollatidoa at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...
# 2
This obsessive setting stuff to null will do you no good. Clearing the vector - or just abandoning the reference to it is fine. You're problem is probably more to do with not closing your Statement after use.
malcolmmca at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...
# 3
@doremifasollatido:I introduced all the useless stuff because calling only _snps.clear did not solve the problem.@malcolmmc:In how far could the statement cause that trouble?
abenstexa at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...
# 4

The trouble is that Connections, Statements, ResultSet etc. when open can hang on to all kinds of resources, and not just in your program but also in the database server.

You should always close them after use with an explicit close() call. Setting the reference to null won't do anything. Get into the habit of doing this in finally clauses so that they get closed even if the code throws an exception.

This may, or may not be the cause of your problem. Fix it and see.

malcolmmca at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...
# 5
Unfortunately no idea really worked. I used jhat to profile the OutOfMemoryError and found that there are over 500000 instances of the class SNP, which is by far more than retrieved with the queries.
abenstexa at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...
# 6

I'd guess, then, that your JOIN is going wrong, perhaps producing the full cartesian product of the tables rather than a single row join for each primary entry. This would give you a numeber of rows equal to the product of the size of the two tables.

You're probably unwise to put the catch statement inside the loop, by the way, if theres an SQLException you should give up.

Test your SQL in a suitable database program like TOAD, SQL+ or whatever comes with the database you're using.

malcolmmca at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...
# 7
I tried the SQL statement already without java and it worked. The strange is that everything works fine the first few times the method performing the query is invoked.
abenstexa at 2007-7-9 5:43:56 > top of Java-index,Java Essentials,Java Programming...