Java Heap Memory Problem!!

The code below is my class.

I read from a file with size of approx. 80Mbytes.

And i get the following error

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

at java.lang.String.concat(String.java:2001)

at Group_Referenced_KWDS_By_Article.main(Group_Referenced_KWDS_By_Articl

e.java:40)

It seems that my tempKWDS gets too long while it shouldn't.

Any proposals?

Tip: My class actually reads from a file and tries to minimize it's length by concatenating lines with the same 1st element of the file. ( ex. ID1 , good ID1 , modest--to--ID1:: good , modest )

import java.io.*;

import java.util.*;

//import java.lang.*;

publicclass Group_Referenced_KWDS_By_Article{

publicstaticvoid main(String[] argz){

try{

String linefeed;

File fin =new File("Keywords Referenced ByArticle.txt");

FileInputStream fis =new FileInputStream(fin);

BufferedReader br =new BufferedReader(new InputStreamReader(fis));

BufferedWriter out =new BufferedWriter(

new OutputStreamWriter(

new FileOutputStream(

new File("Grouped Referenced KWDS ByArticle.txt"))));

Hashtable htable =new Hashtable(100000);

String tempKWDS="";

StringBuffer empty =null;

String currentUT_ITEM ="";

String previousUT_ITEM =null;

String[] line =new String[2];

String tofileValue="";

String tofileKey ="";

String tofile ="";

while((linefeed=br.readLine())!=null){

line = linefeed.split("::");

currentUT_ITEM=line[0];

if(currentUT_ITEM.equalsIgnoreCase(previousUT_ITEM)){

tempKWDS=htable.get(currentUT_ITEM).toString();

tempKWDS = tempKWDS.concat(" :: ");

tempKWDS = tempKWDS.concat(line[1]);//LINE 40 <

//htable.remove(currentUT_ITEM);

htable.put(currentUT_ITEM,tempKWDS);

previousUT_ITEM=currentUT_ITEM;

tempKWDS="";

}else{

tempKWDS ="";

tempKWDS = line[1];

htable.put(line[0],tempKWDS);

previousUT_ITEM=currentUT_ITEM;

}

}

Enumeration k = htable.keys();

while (k.hasMoreElements()){

tofileKey=null;

tofileValue=null;

tofile=null;

tofileKey = k.nextElement().toString();

tofileValue = htable.get(tofileKey).toString();

tofile=tofileKey.concat(" :: ").concat(tofileValue);

out.write(tofile);

out.newLine();

//System.out.println("Key " + key + "; Value " + (String) h.get(key));

}

br.close();

out.close();

}catch(Exception e){

System.out.println(e);

}

}

}

[4468 byte] By [georousa] at [2007-11-26 17:15:28]
# 1

> The code below is my class.

> I read from a file with size of approx. 80Mbytes.

I didn't look at your code but in any case, by default a JVM starts with a maximum of 64MB. Use the -Xmx option to launch with more memory; see

http://java.sun.com/j2se/1.5.0/docs/tooldocs/windows/java.html

Lokoa at 2007-7-8 23:43:28 > top of Java-index,Java Essentials,Java Programming...
# 2
I get the following msg!Invalid maximum heap size: -XmxCould not create the Java virtual machine.How can that be possible?i have JDK and JRE 1.6 installed (wierd)
georousa at 2007-7-8 23:43:28 > top of Java-index,Java Essentials,Java Programming...
# 3

I don't think the following will solve your problem but it gets rid of a load of redundant code.

import java.io.*;

import java.util.*;

public class Group_Referenced_KWDS_By_Article

{

public static void main(String[] argz)

{

try

{

File fin = new File("Keywords Referenced ByArticle.txt");

FileInputStream fis = new FileInputStream(fin);

BufferedReader br = new BufferedReader(new InputStreamReader(fis));

BufferedWriter out = new BufferedWriter(

new OutputStreamWriter(

new FileOutputStream(

new File("Grouped Referenced KWDS ByArticle.txt"))));

Hashtable<String,String> htable = new Hashtable<String,String> (100000);

String previousUT_ITEM = null;

String linefeed;

while((linefeed=br.readLine())!=null)

{

String[] line = linefeed.split("::");

String currentUT_ITEM=line[0];

if(currentUT_ITEM.equalsIgnoreCase(previousUT_ITEM))

{

String tempKWDS = htable.get(currentUT_ITEM);

tempKWDS = tempKWDS.concat(" :: ");

tempKWDS = tempKWDS.concat(line[1]);//LINE 40 <

htable.put(currentUT_ITEM, tempKWDS);

}

else

{

htable.put(line[0], line[1]);

previousUT_ITEM=currentUT_ITEM;

}

}

Enumeration k = htable.keys();

while (k.hasMoreElements())

{

String tofileKey = (String)k.nextElement();

String tofileValue = htable.get(tofileKey);

String tofile=tofileKey.concat(" :: ").concat(tofileValue);

out.write(tofile);

out.newLine();

//System.out.println("Key " + key + "; Value " + (String) h.get(key));

}

br.close();

out.close();

}

catch(Exception e)

{

System.out.println(e);

}

}

}

sabre150a at 2007-7-8 23:43:28 > top of Java-index,Java Essentials,Java Programming...
# 4

Sabre thnx but i still have the mem prob...

How come?

i've read that java -Xmx128m could solve (and did solved) but that is not a solution for somthng serious!! Because once you make something portable you cant tell the user to use compile jvm options!!

I'm a student at http://www.it.teithe.gr and not a commercial programmer

Thnx once again!!

georousa at 2007-7-8 23:43:28 > top of Java-index,Java Essentials,Java Programming...
# 5

> Sabre thnx but i still have the mem prob...

> How come?

> i've read that java could solve (and did

> solved) but that is not a solution for somthng

> serious!! Because once you make something portable

> you cant tell the user to use compile jvm options!!

You are right; changing to use -Xmx128m is not really a solution. I try to never load the whole of a file into memory unless the file is very small. As you have found, it does not scale well.

Why do you need the whole file in memory?

sabre150a at 2007-7-8 23:43:28 > top of Java-index,Java Essentials,Java Programming...
# 6

well, i do not know a way to load partially a file.

my file struct is <aNumber> , <aString>

and currently i have a lot of lines with the same aNumber.

So i try to concat the <aString>s to a single line.

The file is sorted by the means of <aNumber>.

Can you thing of a diff. algorithmic approach?

All i think is that i read per line i process i store to some Hash and at the end i flush my hash to a new file.

i can't find a way to write to my new file partially and "relax" my mem.

Thnx for your help!!

georousa at 2007-7-8 23:43:28 > top of Java-index,Java Essentials,Java Programming...
# 7
If all you are trying to do is create a new file that merges values associated with a key then consider the use of a database instead of a Hashtable.
sabre150a at 2007-7-8 23:43:29 > top of Java-index,Java Essentials,Java Programming...
# 8

Couple options:

1) Use a TreeMap. Not has memory-intensive as a Hashtable/HashMap, but it's slower if you've got a lot of data.

2) Use a file-based hashtable instead of an in-memory hashtable. One solution would be to have one file per hash, but that could also be bad if you have a lot of hashes. The key here would be to balance what you kept in memory and what you kept on disk.

3) Use a database as suggested by sabre150. There are small database options available out there. HSQL I think is one of them.

Jasprea at 2007-7-8 23:43:29 > top of Java-index,Java Essentials,Java Programming...