OutOfMemoryError when Eden space only 1/3 full

The Trouble-Shooting and Diagnostics Guide for JDK 1.5 has a section 2.1 Diagnosing Memory Leaks (http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf) which speaks to the various types of OutOfMemoryErrors that are potentially possible and is worth reading to see if you can narrow down the type of OOME you suffer from.

Your survivor spaces are likely empty because of the GC algorithm you are using. When you start the JVM with the low pause CMS collector, the default SurvivorRatio=1024 and MaxTenuringThreshold=0 (http://java.sun.com/performance/reference/whitepapers/tuning.html section 4.2.6).

Based on your comments, I would guess you are likely not running out of Java Heap but rather running out of native space. You are likely running in a 32-bit JVM given your 4G physical memory but the 3.6G could very well be encroaching the the max process size within your 32-bit JVM. (Alternatively, you might have ran out of swap space given this one process is consuming most of your physical memory.)

Rather then adding more memory, I would monitor your GC logs and decide if your base Java Heap foot print really requires 3G. If it doesn't, I would size your JVM -Xmx smaller to allow more room for your native heap requirements (sockets, threads, stacks, files, nio buffers).

Note:

Adding more memory to your 4G machine will imply switching to the 64-bit JVM which means you'll be able to circumvent the 32-bit pointer barrier but the Java Object sizes all increase. Basically, it could be that if you had 5G of memory allocated you might only have the same amount of Java Objects possible that you do in your 32-bit process. If you are a member of the ACM, http://portal.acm.org/citation.cfm?id=1107406.1107407 is a nice article which discusses the implications of 64-bit over 32-bit JVMs.

drmacdona at 2007-7-15 2:33:36 >

Thanks for all your replies. I haven't got the GC logs any more since I split the process up into smaller chunks to get it to work, but from memory:

- the only flags I set were the Xmx and XX:MaxPermSize

- the survivor spaces were initially used, but towards the time where I get an OOM they become empty and don't fill up

RGibsona at 2007-7-15 2:33:36 >

To come back to this question: I have a related symptom in a small test app (below). My test app just allocates blocks of memory in decreasing chunks until there is no memory left. But on my test machine it fails much earlier that I would have expected - it doesn't manage to allocate 35M when the free memory should be about 140M.

I again used visualgc to look at the heap and again what I don't understand is that my Eden Space is shown as (170.5M, 3.0M):121K which I take to mean that the maximum capacity is 170M but the current capacity is just 3M (of which 121K are used). Why hasn't the current capacity increased to the max capacity? Why does my application throw OOME at this point? How can I realistically estimate the free memory available to my application? Is this just a pathological case that I'm triggering in my test app that I'm unlikely to run up against in production?

Here are the last few lines from the GC logs just before the OOME - at this point my test is trying to allocate 35M and there should be about 140M free. The test code follows and I was running it with JDK 1.6 on an AMD 64-bit machine under RedHat Linux FC6 with java -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -ea -Xmx512M tests.MemoryTest

Thanks a lot!

R{Heap before GC invocations=5 (full 2):

PSYoungGentotal 3584K, used 0K [0x00002aaac9b40000, 0x00002aaaca240000, 0x00002aaad45e0000)

eden space 3072K, 0% used [0x00002aaac9b40000,0x00002aaac9b40000,0x00002aaac9e40000)

from space 512K, 0% used [0x00002aaac9e40000,0x00002aaac9e40000,0x00002aaac9ec0000)

tospace 512K, 0% used [0x00002aaaca1c0000,0x00002aaaca1c0000,0x00002aaaca240000)

PSOldGentotal 349568K, used 318506K [0x00002aaab45e0000, 0x00002aaac9b40000, 0x00002aaac9b40000)

object space 349568K, 91% used [0x00002aaab45e0000,0x00002aaac7ceaad0,0x00002aaac9b40000)

PSPermGentotal 21248K, used 2481K [0x00002aaaaf1e0000, 0x00002aaab06a0000, 0x00002aaab45e0000)

object space 21248K, 11% used [0x00002aaaaf1e0000,0x00002aaaaf44c780,0x00002aaab06a0000)

4.882: [Full GC [PSYoungGen: 0K->0K(3584K)] [PSOldGen: 318506K->318501K(339520K)] 318506K->318501K(343104K) [PSPermGen: 2481K->2478K(21248K)], 0.0143080 secs]

Heap after GC invocations=5 (full 2):

PSYoungGentotal 3584K, used 0K [0x00002aaac9b40000, 0x00002aaaca240000, 0x00002aaad45e0000)

eden space 3072K, 0% used [0x00002aaac9b40000,0x00002aaac9b40000,0x00002aaac9e40000)

from space 512K, 0% used [0x00002aaac9e40000,0x00002aaac9e40000,0x00002aaac9ec0000)

tospace 512K, 0% used [0x00002aaaca1c0000,0x00002aaaca1c0000,0x00002aaaca240000)

PSOldGentotal 339520K, used 318501K [0x00002aaab45e0000, 0x00002aaac9170000, 0x00002aaac9b40000)

object space 339520K, 93% used [0x00002aaab45e0000,0x00002aaac7ce96d0,0x00002aaac9170000)

PSPermGentotal 21248K, used 2478K [0x00002aaaaf1e0000, 0x00002aaab06a0000, 0x00002aaab45e0000)

object space 21248K, 11% used [0x00002aaaaf1e0000,0x00002aaaaf44bac8,0x00002aaab06a0000)

}

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

at com.nexgenfs.apps.tests.memory.MemoryTest.main(MemoryTest.java:24)

Heap

PSYoungGentotal 3584K, used 123K [0x00002aaac9b40000, 0x00002aaaca240000, 0x00002aaad45e0000)

eden space 3072K, 4% used [0x00002aaac9b40000,0x00002aaac9b5ec40,0x00002aaac9e40000)

from space 512K, 0% used [0x00002aaac9e40000,0x00002aaac9e40000,0x00002aaac9ec0000)

tospace 512K, 0% used [0x00002aaaca1c0000,0x00002aaaca1c0000,0x00002aaaca240000)

PSOldGentotal 349568K, used 318501K [0x00002aaab45e0000, 0x00002aaac9b40000, 0x00002aaac9b40000)

object space 349568K, 91% used [0x00002aaab45e0000,0x00002aaac7ce96d0,0x00002aaac9b40000)

PSPermGentotal 21248K, used 2519K [0x00002aaaaf1e0000, 0x00002aaab06a0000, 0x00002aaab45e0000)

object space 21248K, 11% used [0x00002aaaaf1e0000,0x00002aaaaf455f28,0x00002aaab06a0000)

package tests;

import java.util.LinkedList;

import java.util.List;

public class MemoryTest

{

public static void main (String[] args) throws InterruptedException

{

final int periodicity = 1000;

final List<byte[]> fillMemory = new LinkedList<byte[]> ();

while (true)

{

final long mem = getFreeMemory ();

showFree ();

final int take;

if (mem / 4 > (long) Integer.MAX_VALUE)

{

take = Integer.MAX_VALUE;

} else

{

take = (int) (mem / 4);

}

System.out.println ("take = " + (take/1024) + "K");

fillMemory.add (new byte[take]);

System.out.println ("Taken");

showFree ();

Thread.sleep (periodicity);

}

}

private static long getFreeMemory ()

{

// Get total free memory rather than total free committed memory

final Runtime rt = Runtime.getRuntime ();

long free = rt.freeMemory () + rt.maxMemory () - rt.totalMemory ();

return free;

}

private static void showFree ()

{

long free = getFreeMemory ();

System.out.println ("Memory.availableMemory () = " + (free /1024) + "K");

}

}

RGibsona at 2007-7-15 2:33:36 >

Thanks for your hints Jon. I see you mention Xmn in [url http://blogs.sun.com/jonthecollector/entry/the_second_most_important_gc]your blog[/url], I didn't know about it.

My application doesn't actually allocate huge big chunks like my test was, but I managed to solve my problem by refining my understanding of "free space". I changed the code so that "total available space" is defined to be free space in the Eden Space + free + uncommitted space in the Old Generation. (My actual problem was that I wanted to be able to monitor when we were likely to be about to encounter an OutOfMemoryError.) I understand now that it is no good having 100M of uncommitted "available" space in the Young Generation if the Old Generation is full of live objects. Yes I know that's pretty basic, but I had a big "Aha" moment when the penny dropped: there isn't a straightforward mapping from "free" or "uncommitted" to "available for allocation of a new object".... Anyway, armed with this knowledge I was able to rely on the default VM allocation behaviour (i.e. just Xmx, no Xms or Xmn).

Thanks a lot!

R

RGibsona at 2007-7-15 2:33:36 >