OutOfMemoryError, already have set VM arguments

I have to read an extremely large text file (1.16 gb, several million lines)

I am getting an OutOfMemoryError.

I have set VM arguments -Xms1024m -Xmx1024M

As I understand it, setting them to anything greater than these values doesn't have much effect.

What can I do?

staticpublic Vector< String > readOneFile( String filename )

{

Vector< String > vector =new Vector();// data to return

BufferedReader bufferedReader;

String line ="";

String[] lineOfTokens =null;

// read all lines from file

int lineNumber = -1;

try{

BufferedReader br =new BufferedReader(new FileReader( filename ) );

while ( ( line = br.readLine() ) !=null ){

lineNumber++;

vector.add( line );

}// end while still lines to read

}// end try

catch ( FileNotFoundException e ){

System.out.println("FileIO.readDataFile(): " + e +"\n\t\tFile not found: " + filename );

filename =null;

}

catch ( IOException e ){

System.out.println( e +"FileIO.readDataFile(): Problem with file: " + filename );

filename =null;

}

catch ( NullPointerException e ){

System.out.println( e +"FileIO.readDataFile(): Null pointer, file: " + filename );

filename =null;

}

catch ( OutOfMemoryError e ){

System.out.println( e +"FileIO.readDataFile(): OutOfMemoryError, file: " + filename );

System.out.println ("Line number = " + lineNumber);

filename =null;

System.exit(-1);

}

return vector;

}// end()

[2933 byte] By [allelopatha] at [2007-11-27 11:57:54]
# 1

You have to think of a different way of going about what you are trying to do.

Obviously a Vector with several million String objects is just going to be too massive to load into memory all at once.

You have to do it in pieces. Load maybe 10,000 Strings into a Collection, then do what you want with the data and discard it. Then repeat the process getting the next 10,000 lines.

maple_shafta at 2007-7-29 19:16:37 > top of Java-index,Java Essentials,Java Programming...
# 2

> I have to read an extremely large text file (1.16 gb,

> several million lines)

> I am getting an OutOfMemoryError.

> I have set VM arguments -Xms1024m -Xmx1024M

> As I understand it, setting them to anything greater

> than these values doesn't have much effect.

>

> What can I do?

Something else besides pointlessly loading 1.16 GB of a text file into memory.

cotton.ma at 2007-7-29 19:16:37 > top of Java-index,Java Essentials,Java Programming...
# 3

>I have set VM arguments -Xms1024m -Xmx1024M

1.16GB = 1187.84MB < 1024MB

Add in all your other overhead...1024 isn't enough...

but as others have said, why do you need to load that much in at once?

robtafta at 2007-7-29 19:16:37 > top of Java-index,Java Essentials,Java Programming...
# 4

> I have set VM arguments -Xms1024m -Xmx1024M

> As I understand it, setting them to anything greater than these values doesn't have much effect.

Without commenting on the correctness of this approach, you can probably set larger values. Just run a HelloWorld program while trying different values. My setup maxed out at 1322 mb, giving the following at 1323:

java -Xmx1323m HelloWorldApp

Error occurred during initialization of VM

Could not reserve enough space for object heap

Could not create the Java virtual machine.

Each machine's limits can vary, depending on other things in memory.

ChuckBinga at 2007-7-29 19:16:37 > top of Java-index,Java Essentials,Java Programming...
# 5

> > I have set VM arguments -Xms1024m -Xmx1024M

> > As I understand it, setting them to anything

> greater than these values doesn't have much effect.

>

> Without commenting on the correctness of this

> approach,

I think that is a mistake.

Among other problems with the current approach is the fact the OP probably needs closer to 4 GB with the way they are storing this file in memory.

So that's kind of a problem.

cotton.ma at 2007-7-29 19:16:37 > top of Java-index,Java Essentials,Java Programming...
# 6

>>but as others have said, why do you need to load that much in at once?

I don't. I have change the way in which I use the data.

Thanks for all the comments.

allelopatha at 2007-7-29 19:16:37 > top of Java-index,Java Essentials,Java Programming...