Virtaul Machine Hangs

Hi,

Our application is basically http proxy which is running in the jboss enviroment, from time to time, the java process cpu usage spikes to 99%, when this happen the application stops accepting any new connections, it is usually not possible to ask for a thread dump, and when it is possible it shows that 99% of the threads are "Waiting for monitor entry", the only threads which are runnable are the ones that are handing the GC, and a few socket listeners.

for example:

"SocketListener0-16" prio=1 tid=0x08fc5e40 nid=0x24c4 waiting for monitor entry [0x24399000..0x2439a0b0]

When the machine is in hang state, jstat shows that the GC takes around 90 times more than on a machine which is not in an hang state.

The last GC reason appear as Allocation Failed.

It is not possible to use jstack as we are using jdk1.5_06 and this version has a bug which is fixed in jdk1.5_08 (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6359295).

If the process is not restarted it may get out of the hang situation after around 3 hours, but sometimes the cpu usage drops to 0% and the process remains in a "sleep" state in which it doesnt respond to anything until it is restarted.

Unfortunately there is no special scenario to reproduce this behaviour, and it is not related to load, as this happens also during the times of low load.

The java is started with the following properties :

-Xms2g -Xmx2g -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

Any help / hint would be hight appreciated.

Thanks.

[1605 byte] By [Tomer-Cohena] at [2007-11-26 18:05:39]
# 1

Just a guess, but this can be caused if your machine is paging, particularly if the java heap gets paged out and then during GC it must be paged back in. How much memory does the machine have?

The heap size should always be less than the amount of memory on the box (java garbage collectors are not written to deal with heaps that get swapped out to disk). How much less depends on what else you have running, but you have to leave room for the operating system, system libraries, other parts of the JVM, other processes, etc.

jxca at 2007-7-9 5:36:19 > top of Java-index,Java HotSpot Virtual Machine,Specifications...