Out of swap space
Hi All,
I am getting the following exception in my application server
Exception java.lang.OutOfMemoryError: requested 1024000 bytes for GrET* in D:/BUILD_AREA/jdk142-update/ws/fcs/hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
The jvm settings which I use are
-Xms768m -Xmx768m -XX:NewRatio=2 -XX:SurvivorRatio=6
-XX:PermSize=128m -XX:MaxPermSize=128m
OS is windows 2003 server
JDK is 1.4.2 update 9
Previously I was getting this error very frequently (about once in 5 days). After analysis of the GC logs we had set the PermGen to 128mb. With this setting now we are facing the issue once in 20 to 25 days.
Following is the last few sections GC log
{Heap before GC invocations=38175:
Heap
def new generationtotal 229376K, used 196607K [0x10010000, 0x20010000, 0x20010000)
eden space 196608K, 99% used [0x10010000, 0x1c00ffb8, 0x1c010000)
from space 32768K,0% used [0x1e010000, 0x1e010000, 0x20010000)
tospace 32768K,0% used [0x1c010000, 0x1c010000, 0x1e010000)
tenured generationtotal 524288K, used 153684K [0x20010000, 0x40010000, 0x40010000)
the space 524288K, 29% used [0x20010000, 0x29625220, 0x29625400, 0x40010000)
compacting perm gen total 131072K, used 34750K [0x40010000, 0x48010000, 0x48010000)
the space 131072K, 26% used [0x40010000, 0x421ff950, 0x421ffa00, 0x48010000)
1695957.673: [GC 350292K->163450K(753664K), 0.0816114 secs]
Heap after GC invocations=38176:
Heap
def new generationtotal 229376K, used 9765K [0x10010000, 0x20010000, 0x20010000)
eden space 196608K,0% used [0x10010000, 0x10010000, 0x1c010000)
from space 32768K, 29% used [0x1c010000, 0x1c999688, 0x1e010000)
tospace 32768K,0% used [0x1e010000, 0x1e010000, 0x20010000)
tenured generationtotal 524288K, used 153684K [0x20010000, 0x40010000, 0x40010000)
the space 524288K, 29% used [0x20010000, 0x29625220, 0x29625400, 0x40010000)
compacting perm gen total 131072K, used 34750K [0x40010000, 0x48010000, 0x48010000)
the space 131072K, 26% used [0x40010000, 0x421ff950, 0x421ffa00, 0x48010000)
} {Heap before GC invocations=38176:
Heap
def new generationtotal 229376K, used 70639K [0x10010000, 0x20010000, 0x20010000)
eden space 196608K, 30% used [0x10010000, 0x13b828e8, 0x1c010000)
from space 32768K, 29% used [0x1c010000, 0x1c999688, 0x1e010000)
tospace 32768K,0% used [0x1e010000, 0x1e010000, 0x20010000)
tenured generationtotal 524288K, used 153684K [0x20010000, 0x40010000, 0x40010000)
the space 524288K, 29% used [0x20010000, 0x29625220, 0x29625400, 0x40010000)
compacting perm gen total 131072K, used 34752K [0x40010000, 0x48010000, 0x48010000)
the space 131072K, 26% used [0x40010000, 0x42200330, 0x42200400, 0x48010000)
1695971.695: [Full GC
The GC seems to indicate that I have sufficient space left in jvm heap as only 29% is used in tenured generation. What could be the source of this Out of swap space error.
Any help is greatly appreciated.
Thanks & Regards,
Ramkumar.
# 1
Might want to increase the amount of memory you give the JVM. Or use less in the program.Also, make sure you don't have any memory leaks.
# 2
> Might want to increase the amount of memory you give
> the JVM. Or use less in the program.
>
> Also, make sure you don't have any memory leaks.
Already my tenured generation is showing only 29% as the used heap the rest 71% is free. So how would increasing the JVM heap settings going to help
# 3
> > Might want to increase the amount of memory you
> give
> > the JVM. Or use less in the program.
> >
> > Also, make sure you don't have any memory leaks.
>
> Already my tenured generation is showing only 29% as
> the used heap the rest 71% is free. So how would
> increasing the JVM heap settings going to help
Well you posted that you got a:
java.lang.OutOfMemoryError
# 4
> > > Might want to increase the amount of memory you
> > give
> > > the JVM. Or use less in the program.
> > >
> > > Also, make sure you don't have any memory leaks.
> >
> > Already my tenured generation is showing only 29%
> as
> > the used heap the rest 71% is free. So how would
> > increasing the JVM heap settings going to help
>
> Well you posted that you got a:
>
> java.lang.OutOfMemoryError
If you look, he has plenty of heap. The problem is that the hotspot compiler is asking the OS for more memory outside the heap and for some reason the WIndows OS is refusing to grant it more memory. There are several things that use memory outside of the heap. Native (JNI) methods can directly allocate memory from the OS. NIO uses memory directly, hotspot compiler, and there are others. Find out how much memory your application is taking up according to Windows. You actually might have better success lowering the heap and perm. Also you can try 64bit windows or another 64bit OS. Windows is know not to let you use much of the 32bit of memory space. You can allocate a lot more memory on most 32bit UNIX type OS than on Windows.
# 5
I would also suggest that you use the latest update of java for 1.4.
# 6
> I would also suggest that you use the latest update
> of java for 1.4.
Its a production system and updating it might take weeks/months (need to first test all in test environment and all system compatiblity and then move to production) also updating to latest update should have a valid reason and will this issue not come in the latest update.
Is there any other way by which we can solve this one, like increasing of swap space or lowering of JVM heap, etc.,
what are the other possibilities...
# 7
Take a look at the [url http://java.sun.com/j2se/1.4.2/ReleaseNotes.html]Release Notes for 1.4.2[/url]. It contains all the bug fixes since the release you are using. The second listed bug fix for [url http://java.sun.com/j2se/1.4.2/ReleaseNotes.html#142_13]1.4.2_13[/url], [url http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6420685]6420685[/url] mentions something about growable array. I don't know if it applies. You should check all the bug fixes and see if any of them could cause critical problems.
# 8
>
> Previously I was getting this error very frequently
> (about once in 5 days). After analysis of the GC logs
> we had set the PermGen to 128mb. With this setting
> now we are facing the issue once in 20 to 25 days.
> ...
>
> The GC seems to indicate that I have sufficient space
> left in jvm heap as only 29% is used in tenured
> generation. What could be the source of this Out of
> swap space error.
>
I suspect you will find that if you lower the perm gen space that the error will occur much more often.
And that would tend to demonstrate the real likelyhood that something in your code is growing without bound either by need or due to incorrect coding. You would then have to find the leak and fix it. Or just bounce the server often enough that it isn't ever a problem.
# 9
> I suspect you will find that if you lower the perm
> gen space that the error will occur much more often.
>
> And that would tend to demonstrate the real
> likelyhood that something in your code is growing
> without bound either by need or due to incorrect
> coding. You would then have to find the leak and fix
> it. Or just bounce the server often enough that it
> isn't ever a problem.
I don't think that the error I got is due to leak on the application code.
First the error states that it requested 1024000 bytes but could not allocate due to out of swap space.
Second the GC log just before the full GC which gave the exception states that it has sufficient space(71 % is free in tenured and 74% is free in permgen). In these cases why is hotspot not finding 1MB for allocation.
I can even send you the full GC log (which is for 20 days around 2.7MB after zip) which would clearly establish that the application heap usage does not keep increasing.
# 10
>
> I don't think that the error I got is due to leak on
> the application code.
>
You decreased the failure rate by increasing the heap size. That is a standard scenario for a leak.
So it is either a leak in the VM or a leak in your code. Reducing, rather than decreasing the perm size is going to increase the failure rate and that should make it easier for you to detemine the cause.
# 11
>
> You decreased the failure rate by increasing the heap
> size. That is a standard scenario for a leak.
>
Actually I increased the PermSize from 32mb to 128mb but at the same time I had decreased the -xms and -xmx values from 1GB to 768mb (reduction of 256mb).
We had increased the PermSize because the previous GC logs showed 99% usage. But now the PermSize is showing only 26% usage and tenured is showing only 29% usage. Still the hotSpot is throwing exception saying it cannot allocate 1MB which is strange.
Meanwhile as Caffeine0001 suggested I had a look at the Release Notes for 1.4.2. It does have some critical fixes for OOM. Some of them are 4855795, 6186200, 6280181.
# 12
> >
> > You decreased the failure rate by increasing the
> heap
> > size. That is a standard scenario for a leak.
> >
jschell is right almost all the time, but this time he is wrong. For java coded memory leaks that fill up the heap because the application was not releasing references, jschell is correct. This isn't a standard OOM do to the heap being full. This OOM is because of total process memory address space exhaustion. If this is a native memory leak. It is a C, C++ native type where a reference was lost to memory. Either in the VM or a native dll. You decrease the failure rate by decreasing the heap. See my explanation below.
>
> Actually I increased the PermSize from 32mb to 128mb
> but at the same time I had decreased the -xms and
> -xmx values from 1GB to 768mb (reduction of 256mb).
> We had increased the PermSize because the previous GC
> logs showed 99% usage. But now the PermSize is
> showing only 26% usage and tenured is showing only
> 29% usage. Still the hotSpot is throwing exception
> saying it cannot allocate 1MB which is strange.
>
The type of OOM your getting has nothing to do with the percentage of perm or heap you are using. At times the VM will request memory outside of the heap and perm. When it needs this other memory, it will request the memory directly from the OS using a malloc call. The OS then gives it the memory. For some reason Windows 2003 is refusing to give VM 1MB more memory.
Now the question is why won't Windows 2003 give the VM more memory when it does a malloc?
Normally a Windows process are limited to 2G of total memory under Windows OS. There is a Windows 2003 OS setting that lets you get 3G. I'm not sure if Sun's VM can take advantage of this switch. I do know that it won't let you get larger heaps because of where windows bases some of its dlls in memory. See http://www.unixville.com/~moazam/
Let's assume you haven't used that switch so the total memory your process can use is 2G. This includes heap(768MB) + perm(128MB) + code cache(?MB) + malloc memory(?MB). You can use jconsole to see the code cache memory usage. Looking at one of my java programs it shows the code cache using 50M. The malloc memory is used for a lot of things and can grow quite large. What complicates things further is the windows loads some of its dlls in the 2G segment so you really don't have 2G all to yourself. If all 5 memory totals go beyond 2G or you total swap space, which ever is smaller, you will get a OOM.
Now we know 768MB(heap) + 128MB(perm) + ~50MB(code cache) + ~256MB(Windows dlls - just a guess here) = ~1202MB. You'll get more accurate results if you use step 1 below at the time your application starts up. I'll assume your swap file is at least 2GB so your not really running out of swap space. This give you 2048MB - ~1202MB = ~846MB. This 846MB is what is what you are running out off. If you use the 3G switch you could have 1870MB. The 846MB is then used on demand by the VM for compiling, nio, etc. And native dlls can also do their own mallocs and use up this memory. As an example of this, at one time I was using Oracle's OCI JDBC driver and it alone was using more than 1GB of native memory and was giving us OOM when our app approached 4GB on a Sun box. That is why I switch to the thin driver.
Depending on your application, even without a memory leak by some native dll or the vm itself, 846MB may not be enough memory.
Suggestions:
1. See what the total VM memory usage as reported by Windows OS 2003. You can use ctrl-shift-escape. Click on the processes tab and then got to the menu: View->Select Columns... and add VM memory column. This will let you know how much memory you are using and if you are getting close to the OOM.
2. Lower your heap and/or perm. This will increase the ~846MB you have left over for native memory used by native malloc calls.
3. Increase the total process memory address space. Make sure you swap file is big enough. Try using the [url http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx]Windows 3g switch[/url]. I'm not sure if Sun's VM can use it though. Or switch to a different OS. Use 64bit Windows OS and the 64Bit VM. It doesn't have the 2G limit. Most Unix 32bit OS can use almost 4G.
4. If the VM is leaking memory then all you can do is try a newer version. If your are using native dlls, then I would check them for memory leaks or a newer versions.
# 13
> > >
> > > You decreased the failure rate by increasing the
> > heap
> > > size. That is a standard scenario for a leak.
> > >
>
> jschell is right almost all the time, but this time
> he is wrong. For java coded memory leaks that fill up
> the heap because the application was not releasing
> references, jschell is correct. This isn't a
> standard OOM do to the heap being full. This OOM is
> because of total process memory address space
> exhaustion. If this is a native memory leak. It is
> a C, C++ native type where a reference was lost to
> memory. Either in the VM or a native dll. You
> decrease the failure rate by decreasing the
> heap. See my explanation below.
Correct. All I saw was the OOM itself and I didn't look at the text following it.
> 3. Increase the total process memory address space.
> Make sure you swap file is big enough. Try using the
> Windows 3g switch. I'm not sure
> if Sun's VM can use it though.
As far as I can tell via reseach without attempting it myself the VM still limits it.
# 14
Generally speaking, there are usually two reasons on why you'll get an OOME when you have apparently plenty of heap space left:
1) You suffer from memory fragmentation. So, while there is apparently enough heap left, there is no enough contiguous space to allocate what is needed.
2) Space is needed outside the JVM (Caffeine's point). Specially true if you're using JNI.
# 15
> Generally speaking, there are usually two reasons on
> why you'll get an OOME when you have apparently
> plenty of heap space left:
>
> 1) You suffer from memory fragmentation. So, while
> there is apparently enough heap left, there is no
> enough contiguous space to allocate what is
> needed.
Sun's VM can move heap memory around so Java heap fragmentation can be taken care of by a Full GC.
> 2) Space is needed outside the JVM (Caffeine's
> point). Specially true if you're using JNI.
You did make a good point about memory fragmentation. The memory outside the VM's control could become fragmented and would be another reason the OS may refuse granting memory to JNI or the VM itself even without reaching the 2G windows barrier. This can be also be minimized by reducing the heap and therefore having more memory available outside the heap.
