Out of swap space
Hello,
Sun Application Server 8.1 EE Q2 crashes with the following erreur in the production site.
Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 1547216 bytes for Chunk::new. Out of swap space?
Working in Solaris 10 V20z (bi-processeur) with 4GB RAM and 8GB swap. Load averages of the serveur not exceeded to 0.08.
In JVM parameters, i have -Xmx3500m and -XX:MaxPermSize=128m.
Could someone tell me, how can fixe this problem?
Regards,
Thangamathi
It looks to me like you've run out of process virtual address space. You have a few choices on where to go next:
1: reduce your thread stack size with -Xss. This can regain some process address space if you have a sufficient number of threads. The default is platform dependent, but I think it's 1m for your platform. Try 512k, 256k, or even 128k. The value that's best for your app will depend on the application. Too small a value and you'll be getting stack overflow exceptions.
2: reduce your -Xmx value. This may have other performance consequences, as a smaller heap may result in more frequent garbage collections. Some young gen size tuning may overcome those issues, though.
3: use the 64-bit JVM. Just add -d64 to your command line. This too can have performance consequences. You'll need to measure both 32-bit and 64-bit to determine what those consequence are.
HTH
Brian
Go to
http://java.sun.com/javase/
and look in the right nav area for a link named Trouble-Shooting Guide. This
is the JDK 5.0 Troubleshooting guide. Page 167 contains some info about your
question:
Exception in thread 搈ain?java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space?
Although it appears that an OutOfMemoryError is thrown this apparent exception is reported by the HotSpot VM code when an allocation from the native heap failed and the native heap may be close to exhaustion. The message indicates the size (in bytes) of the request that failed and also indicates what the memory is required for. In some cases the reason will be shown but in most cases the reason will be the name of a source module reporting the allocation failure. If an OutOfMemoryError with this error is thrown it may require using utilities on the operating system to diagnose the issue further. Examples of issues that may not be related to the application are when the operating system is configured with insufficient swap space, or when there is another process on the system that is consuming all memory resources. If neither of these issues is the cause then it is possible that the application is failed due to native leak; for example, application or library code is continuously allocating memory but is not releasing it to the operating system.
HTH
Hi,
Thanks for your informations.
I changed my JVM settings like -Xms2048m -Xmx2048m and -Xss512k
Even these modifications, I got many "java.lang.OutOfMemoryError: unable to create new native thread" exceptions.I got these exceptions since I have added 2 more domains in the same production server. Each domain has 2 applications.
Maybe the problem comes over loaded -Xmx value? because the sum of 3 domain's -Xmx value is greater than 4GB (RAM memory)?
Below is the System information :
load averages: 0.03, 0.03, 0.03
41 processes: 40 sleeping, 1 on cpu
CPU states: 98.8% idle, 0.7% user, 0.5% kernel, 0.0% iowait, 0.0% swap
Memory: 4031M real, 2385M free, 2554M swap in use, 6712M swap free
- Server. log --
[#|2006-06-23T13:05:37.890+0200|WARNING|sun-appserver-ee8.1_02|javax.enterprise.system.stream.err|_ThreadID=35;|
Exception in thread "Timer-2" |#]
[#|2006-06-23T13:05:37.891+0200|WARNING|sun-appserver-ee8.1_02|javax.enterprise.system.stream.err|_ThreadID=35;|java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$6.run(ThreadPoolImpl.java:257)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl.createWorkerThread(ThreadPoolImpl.java:241)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl.notifyForAvailableWork(ThreadPoolImpl.java:205)
at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.addWork(WorkQueueImpl.java:99)
at com.sun.ejb.containers.util.ContainerWorkPool.addLast(ContainerWorkPool.java:57)
at com.sun.ejb.containers.util.ContainerWorkPool.addLast(ContainerWorkPool.java:42)
at com.sun.ejb.containers.util.pool.NonBlockingPool$PoolResizeTimerTask.run(NonBlockingPool.java:594)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
|#]
[#|2006-06-23T13:06:35.662+0200|WARNING|sun-appserver-ee8.1_02|javax.enterprise.system.stream.err|_ThreadID=36;|
Exception in thread "RMI RenewClean-[192.168.3.74:49110,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory@101f417]" |#]
[#|2006-06-23T13:06:35.662+0200|WARNING|sun-appserver-ee8.1_02|javax.enterprise.system.stream.err|_ThreadID=36;|java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:322)
at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
at java.lang.Thread.run(Thread.java:595)
I'm pretty sure the problem is that -Xss is specified in KB, so that when you use -Xss512k, you are asking for each thread stack to reserve 512MB (512KB * 1024) of address space. You won't get many of those in a 32-bit address space. Use -Xss512 to request 512KB stacks, though that's the default size for thread stacks, so you would get the same effect by omitting the -Xss parameter. If you want to try smaller thread stacks, try -Xss256 or -Xss128, but watch out for stack overflow errors, which will occur if you need more stack space than the limit you've imposed.
Meanwhile, you should be able to monitor your swap space usage with vmstat.
I mis-spoke: while thread stack sizes are internally kept in KB's, the command line parsing for -Xss takes that into account. So your use of -Xss512k is fine, and my suggestion to use -Xss512 will result in an error message. Sorry for the confusion.
Thanks Peter. I juste changed my Xss configuration into -Xss256k. Our server have 64 bit. But I don't know, I should change into 512k or can I leave 256k? ThanksMathi
If you can run in a 32-bit JVM, your performance will almost certainly be better than a 64-bit JVM (memory bandwidth, cache efficiency, etc.). If you aren't getting OutOfMemoryErrors or StackOverflowErrors, then you should stick with what you have.
How close are you to running out of space in the Java object heap (the region specied by -Xms -Xmx)? Running with -XX:+PrintGCDetails will tell you. It will print lines like [Full GC [PSYoungGen: 128K->0K(10752K)] [PSOldGen: 0K->105K(24576K)] 128K->105K(35328K) [PSPermGen: 1618K->1618K(16384K)], 0.0374620 secs]
where you want to look at the part that corresponds to "128K->105K(35328K)" to show the size of the heap before a full collection, after a full collection, and the size of the heap. (This is just an example: your numbers will be larger because of your -Xms setting and because you are running a real application.) Use the "after" number to adjust the size of your -Xms and -Xmx, according to the GC tuning guide at http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html.
How many threads do you have? Sending a SIGQUIT to the JVM will generate a thread dump, or use prstat(1M) to see the number of LWP's in the java process. If you have a lot of threads (100's or 1000's) and don't need the default stack size, you should consider reducing your -Xss. (As you have discovered.)
You could try running pmap(1) and trying to identify other large regions of your process address space, e.g., C malloc space used by native libraries, etc. But that's often difficult and unproductive.
Hi Peter, thanks for your helpful informations. Since I modified the JVM configurations as I specified. I do not have any more "OutOfMemory" error. Best Regards, Mathi
Hi Peter,
Our server application is running RedHad Linux on HP machine with dual cpu and 2gb ram.
I am working on GC tunning. In that process, I am using following Memory options but application got crashed with error.
javaoptions used:: -Xmx1024m -Xms1024m -XX:NewSize=448m -XX:MaxNewSize=448m -XX:SurvivorRatio=6 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
application crashed with following error::
49570.827: [GC [PSYoungGen: 383994K->26497K(418496K)] 576248K->226305K(1008320K), 0.1164240 secs]
49570.944: [Full GC
Exception java.lang.OutOfMemoryError: requested 128000 bytes for GrET* in /BUILD_AREA/jdk1.5.0_05/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?
Can you pls.. help me out.
Thanks in advance.
VenuGujjari
HiThe application didnt crash... it just complained OOME.You are probably running out of C-heap or really as suggested, swap space. You may want to lower the -Xmx and of cos ur newsize setting and see if u can avoid this error.
Hello,I just opened"Hotspot leaks memory on smp linux", http://forum.java.sun.com/thread.jspa?threadID=5190830
