Thread "waiting on condition" without indicating what the condition is

Does anyone from the Sun HotSpot team read this?

To clarify my question:

Usually the HotSpot thread dump indicates what Java object a blocked thread is waiting on. In my example, "Thread-6" is waiting to lock <0x650f15f8>.

However, sometimes there is no indication of what is being waited on. In my example, "QueueRequestScheduled" has no such indication.

Further, the top of the stack for "QueueRequestScheduled" (AbstractStringBuilder.java:628) is the first line of AbstractStringBuilder.append() which does nothing capable of blocking. (And the method is not synchronized.) How could this thread be blocked at this point?

So, can someone familiar with HotSpot internals please tell me:

What are the possible conditions which HotSpot can indicate a thread is "waiting on condition"?

Why would the top of stack indicate the first line in a (non-synchronized) method? (E.g. does this imply it was actually interrupted by some JVM event while executing elsewhere in this method?)

(I will note that the AbstractStringBuilder.java:628 does reference a static member on Long. So it could be blocked waiting for the Long class to load. But I find it hard to believe that Long wasn't already loaded - this method has executed successfully before many times in this JVM instance - or that it something has wedged loading the Long class - it's hung in this state.)

DavidLennerta at 2007-7-14 19:25:23 >

Thanks very much for the reply.

Is there a way for me to get a C stack trace?

The full HotSpot stack dump for this failure is:

Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode, sharing):

"Thread-6" prio=1 tid=0x083a0480 nid=0x53d waiting for monitor entry [0x4732e000..0x4732ec9c]

at com.thetus.common.util.BytesOut.returnBytesOut(BytesOut.java:207)

- waiting to lock <0x650efdf0> (a java.lang.Object)

- locked <0x650f15f8> (a com.thetus.common.util.BytesOut)

at com.thetus.common.requestservice.bytetransport.SessionByteClient.closeRequestService(SessionByteClient.java:569)

- locked <0x650bc568> (a java.lang.Object)

at com.thetus.common.requestservice.bytetransport.ResponseListener.run(SessionByteClient.java:1532)

at java.lang.Thread.run(Thread.java:595)

"WT:rest:4" prio=1 tid=0x080fffb8 nid=0x45c in Object.wait() [0x471ae000..0x471aef1c]

at java.lang.Object.wait(Native Method)

- waiting on <0x65179060> (a java.lang.Object)

at java.lang.Object.wait(Object.java:474)

at com.thetus.platform.util.ThreadPoolInst$WorkerThread.run(ThreadPoolInst.java:613)

- locked <0x65179060> (a java.lang.Object)

at java.lang.Thread.run(Thread.java:595)

"WT:rest:3" prio=1 tid=0x0807ab20 nid=0x1ed in Object.wait() [0x46f7b000..0x46f7bf9c]

at java.lang.Object.wait(Native Method)

- waiting on <0x65110420> (a java.lang.Object)

at java.lang.Object.wait(Object.java:474)

at com.thetus.platform.util.ThreadPoolInst$WorkerThread.run(ThreadPoolInst.java:613)

- locked <0x65110420> (a java.lang.Object)

at java.lang.Thread.run(Thread.java:595)

"RequestByteListener" prio=1 tid=0x0854be18 nid=0x1e9 runnable [0x4712e000..0x4712ec9c]

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x650ee9b8> (a java.net.SocksSocketImpl)

at java.net.ServerSocket.implAccept(ServerSocket.java:450)

at java.net.ServerSocket.accept(ServerSocket.java:421)

at com.thetus.server.common.requestservice.bytetransport.RequestByteListener.run(RequestByteListener.java:237)

at com.thetus.platform.util.ThreadPoolInst$WorkerThread.run(ThreadPoolInst.java:599)

at java.lang.Thread.run(Thread.java:595)

"QueueRequestScheduled" prio=1 tid=0x085622f8 nid=0x1e8 waiting on condition [0x470ae000..0x470af09c]

at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:628)

at java.lang.StringBuilder.append(StringBuilder.java:222)

at com.thetus.common.util.BytesOut.logCacheStats(BytesOut.java:84)

- locked <0x650efdf0> (a java.lang.Object)

at com.thetus.server.common.requestListener.ListenerServiceMgr.lclLogStats(ListenerServiceMgr.java:259)

at com.thetus.server.common.context.AContextService.logStats(AContextService.java:224)

at com.thetus.server.common.context.AContext.logStats(AContext.java:1633)

at com.thetus.server.common.handler.LclLogStatsHandler.executeRequest(LclLogStatsHandler.java:64)

at com.thetus.server.common.requestservice.RequestAsyncHandlerChain.execute(RequestAsyncHandlerChain.java:96)

at com.thetus.server.common.request.LclLogStats.execute(LclLogStats.java:67)

at com.thetus.server.common.requestPump.QueueRequestScheduled.run(QueueRequestScheduled.java:188)

at com.thetus.platform.util.ThreadPoolInst$WorkerThread.run(ThreadPoolInst.java:599)

at java.lang.Thread.run(Thread.java:595)

"QueueRequestFifo" prio=1 tid=0x085628b8 nid=0x1e7 in Object.wait() [0x4702e000..0x4702f09c]

at java.lang.Object.wait(Native Method)

- waiting on <0x650eec70> (a com.thetus.server.common.requestPump.QueueRequestFifo)

at java.lang.Object.wait(Object.java:474)

at com.thetus.server.common.requestPump.QueueRequestFifo.run(QueueRequestFifo.java:151)

- locked <0x650eec70> (a com.thetus.server.common.requestPump.QueueRequestFifo)

at com.thetus.platform.util.ThreadPoolInst$WorkerThread.run(ThreadPoolInst.java:599)

at java.lang.Thread.run(Thread.java:595)

"DestroyJavaVM" prio=1 tid=0x0805cd88 nid=0x1db waiting on condition [0x00000000..0xbfffbde0]

"Wrapper-Connection" prio=1 tid=0x084fdd98 nid=0x1e4 runnable [0x46ee6000..0x46ee709c]

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at java.net.SocketInputStream.read(SocketInputStream.java:182)

at java.io.DataInputStream.readByte(DataInputStream.java:241)

at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:2348)

at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:2696)

at java.lang.Thread.run(Thread.java:595)

"Wrapper-Control-Event-Monitor" daemon prio=1 tid=0x084fd908 nid=0x1e3 sleeping[0x46e66000..0x46e6709c]

at java.lang.Thread.sleep(Native Method)

at org.tanukisoftware.wrapper.WrapperManager$2.run(WrapperManager.java:701)

"Low Memory Detector" daemon prio=1 tid=0x080a39c0 nid=0x1e1 runnable [0x00000000..0x00000000]

"CompilerThread0" daemon prio=1 tid=0x080a22e8 nid=0x1e0 waiting on condition [0x00000000..0x417afa84]

"Signal Dispatcher" daemon prio=1 tid=0x080a1428 nid=0x1df waiting on condition [0x00000000..0x00000000]

"Finalizer" daemon prio=1 tid=0x0809a890 nid=0x1de in Object.wait() [0x4147d000..0x4147d49c]

at java.lang.Object.wait(Native Method)

- waiting on <0x64f05a60> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)

- locked <0x64f05a60> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x08099b90 nid=0x1dd in Object.wait() [0x413fd000..0x413fd49c]

at java.lang.Object.wait(Native Method)

- waiting on <0x64f05ae0> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:474)

at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

- locked <0x64f05ae0> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=1 tid=0x08097058 nid=0x1dc runnable

"VM Periodic Task Thread" prio=1 tid=0x080be850 nid=0x1e2 waiting on condition

DavidLennerta at 2007-7-14 19:25:23 >

Thanks for the pointer to 6401751.

We also just noticed 6298877 which sounds similar and relates the problem to particular versions of Red Hat Linux (versions which our failing customer site is also running). However the 6298877 symptoms are slightly different from what we experience. (We can ususally obtain HotSpot stack dumps via SIGQUIT, although sometimes not.)

So we'll probably try to get them to upgrade their Linux and see what effect that has. Otherwise we'll see if we can have them install the debug JDK binaries.

Thanks very much for all your help!

DavidLennerta at 2007-7-14 19:25:23 >

I have a similar problem:

using:

java:

java version "1.5.0_11"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)

Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_11-b03, mixed mode)

Linux version and release:

Linux version 2.6.9-42.0.10.ELsmp (brewbuilder@ls20-bc1-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)) #1 SMP Fri Feb 16 17:13:42 EST 2007

Red Hat Enterprise Linux ES release 4 (Nahant Update 4)

There happens a couple of things remarkable:

- memory; max set to 1024M, htop shows RES 1007M, VIRT 1404M. however no "OutOfMemory" messages or anything related seem to happen.

- This happens after many hours of a process - which in other machines has worked well.

===============

thread dump:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_11-b03 mixed mode):

"RedirectionResolverQueue" daemon prio=1 tid=0x0000002ae894a990 nid=0x5437 in Object.wait() [0x0000000041668000..0x0000000041668cb0]

at java.lang.Object.wait(Native Method)

- waiting on <0x0000002ab2dd5fe0> (a com.properazzi.mutch.redirection.RedirectionResolverQueue)

at java.lang.Object.wait(Object.java:474)

at com.properazzi.mutch.redirection.RedirectionResolverQueue.run(RedirectionResolverQueue.java:70)

- locked <0x0000002ab2dd5fe0> (a com.properazzi.mutch.redirection.RedirectionResolverQueue)

"QueryResolver" prio=1 tid=0x0000002ae4e7a320 nid=0x5436 waiting on condition [0x0000000041567000..0x0000000041567d30]

at java.lang.Thread.sleep(Native Method)

at java.lang.Thread.sleep(Thread.java:276)

at com.properazzi.mutch.QueryResolverQueue.run(QueryResolverQueue.java:61)

"Thread-7" prio=1 tid=0x0000002ae89f8aa0 nid=0x5435 runnable [0x0000000041466000..0x0000000041466db0]

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

- locked <0x0000002ab3184cf8> (a java.net.SocksSocketImpl)

at java.net.ServerSocket.implAccept(ServerSocket.java:450)

at java.net.ServerSocket.accept(ServerSocket.java:421)

at com.properazzi.mutch.monitor.Monitor.run(Monitor.java:67)

"Worker0" prio=1 tid=0x0000002ae7850500 nid=0x5434 in Object.wait() [0x0000000041365000..0x0000000041365e30]

at java.lang.Object.wait(Native Method)

- waiting on <0x0000002aa19013f0> (a com.properazzi.mutch.resource.ResourcePool)

at java.lang.Object.wait(Object.java:474)

at com.properazzi.mutch.resource.ResourcePool.getToProcess(ResourcePool.java:116)

- locked <0x0000002aa19013f0> (a com.properazzi.mutch.resource.ResourcePool)

at com.properazzi.mutch.Crawler.getQueryToProcess(Crawler.java:357)

at com.properazzi.mutch.Worker.run(Worker.java:36)

"NB Reader 1" prio=1 tid=0x0000002ae1d771e0 nid=0x5431 waiting on condition [0x0000000041264000..0x0000000041264eb0]

at sun.misc.Unsafe.park(Native Method)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)

at com.properazzi.nioserver.utils.workerThread.run(workerThread.java:46)

"Connection Monitor" prio=1 tid=0x0000002ae5222c00 nid=0x5430 in Object.wait() [0x0000000041163000..0x0000000041163b30]

at java.lang.Object.wait(Native Method)

at com.properazzi.nioserver.PersistentNetSession$Monitor.run(PersistentNetSession.java:82)

- locked <0x0000002aa195dc40> (a com.properazzi.nioserver.PersistentNetSession$Monitor)

"NetOutputQueue" daemon prio=1 tid=0x0000002ae75f58e0 nid=0x542f waiting on condition [0x0000000041062000..0x0000000041062bb0]

at sun.misc.Unsafe.park(Native Method)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)

at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)

at com.properazzi.nioserver.NetOutputQueue.run(NetOutputQueue.java:143)

at java.lang.Thread.run(Thread.java:595)

"Thread-3" prio=1 tid=0x0000002ae75f5090 nid=0x542e runnable [0x0000000040f61000..0x0000000040f61c30]

at sun.nio.ch.PollArrayWrapper.poll0(Native Method)

at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)

at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

- locked <0x0000002aa195e8d0> (a sun.nio.ch.Util$1)

- locked <0x0000002aa195e8b8> (a java.util.Collections$UnmodifiableSet)

- locked <0x0000002aa195e530> (a sun.nio.ch.PollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

at com.properazzi.nioserver.core.NIONetHandler.selectLoop(NIONetHandler.java:302)

at com.properazzi.nioserver.core.NIONetHandler.access$100(NIONetHandler.java:39)

at com.properazzi.nioserver.core.NIONetHandler$2.run(NIONetHandler.java:95)

"Low Memory Detector" daemon prio=1 tid=0x0000002ae1d71130 nid=0x542c runnable [0x0000000000000000..0x0000000000000000]

"CompilerThread1" daemon prio=1 tid=0x0000002ae1d6f500 nid=0x542b waiting on condition [0x0000000000000000..0x0000000040c5d6e0]

"CompilerThread0" daemon prio=1 tid=0x0000002ae1d6df50 nid=0x542a waiting on condition [0x0000000000000000..0x0000000040b5c760]

"AdapterThread" daemon prio=1 tid=0x0000002ae1d6c7f0 nid=0x5429 waiting on condition [0x0000000000000000..0x0000000000000000]

"Signal Dispatcher" daemon prio=1 tid=0x0000002ae1d6b4f0 nid=0x5428 waiting on condition [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=1 tid=0x0000002ae1d57100 nid=0x5427 in Object.wait() [0x000000004085a000..0x000000004085abb0]

at java.lang.Object.wait(Native Method)

- waiting on <0x0000002aa1951c68> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)

- locked <0x0000002aa1951c68> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x0000002ae1d568d0 nid=0x5426 in Object.wait() [0x0000000040759000..0x0000000040759c30]

at java.lang.Object.wait(Native Method)

- waiting on <0x0000002aa193de10> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:474)

at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

- locked <0x0000002aa193de10> (a java.lang.ref.Reference$Lock)

"Crawler" prio=1 tid=0x0000000040117fb0 nid=0x5420 waiting on condition [0x0000007fbfffb000..0x0000007fbfffc420]

at java.lang.Thread.sleep(Native Method)

at java.lang.Thread.sleep(Thread.java:276)

at com.properazzi.mutch.Crawler.run(Crawler.java:406)

at com.properazzi.crawl.crawlers.template.AbstractTemplateCrawler.crawl(AbstractTemplateCrawler.java:168)

at com.properazzi.crawl.crawlers.template.AsyncTemplateCrawler.main(AsyncTemplateCrawler.java:72)

"VM Thread" prio=1 tid=0x0000002ae1d523d0 nid=0x5425 runnable

"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040135730 nid=0x5421 runnable

"GC task thread#1 (ParallelGC)" prio=1 tid=0x00000000401365b0 nid=0x5422 runnable

"GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040137430 nid=0x5423 runnable

"GC task thread#3 (ParallelGC)" prio=1 tid=0x00000000401382b0 nid=0x5424 runnable

"VM Periodic Task Thread" prio=1 tid=0x0000002ae1d72d80 nid=0x542d waiting on condition

===============

pstack:

Thread 22 (Thread 1076189536 (LWP 21537)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a57 in os::Linux::safe_cond_wait ()

#2 0x0000002a95acac39 in Monitor::wait ()

#3 0x0000002a95895143 in GCTaskManager::get_task ()

#4 0x0000002a95896213 in GCTaskThread::run ()

#5 0x0000002a95ae3615 in _start ()

#6 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#7 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#8 0x0000000000000000 in ? ()

Thread 21 (Thread 1077242208 (LWP 21538)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a57 in os::Linux::safe_cond_wait ()

#2 0x0000002a95acac39 in Monitor::wait ()

#3 0x0000002a95895143 in GCTaskManager::get_task ()

#4 0x0000002a95896213 in GCTaskThread::run ()

#5 0x0000002a95ae3615 in _start ()

#6 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#7 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#8 0x0000000000000000 in ? ()

Thread 20 (Thread 1078294880 (LWP 21539)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a57 in os::Linux::safe_cond_wait ()

#2 0x0000002a95acac39 in Monitor::wait ()

#3 0x0000002a95895143 in GCTaskManager::get_task ()

#4 0x0000002a95896213 in GCTaskThread::run ()

#5 0x0000002a95ae3615 in _start ()

#6 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#7 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#8 0x0000000000000000 in ? ()

Thread 19 (Thread 1079347552 (LWP 21540)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a57 in os::Linux::safe_cond_wait ()

#2 0x0000002a95acac39 in Monitor::wait ()

#3 0x0000002a95895143 in GCTaskManager::get_task ()

#4 0x0000002a95896213 in GCTaskThread::run ()

#5 0x0000002a95ae3615 in _start ()

#6 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#7 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#8 0x0000000000000000 in ? ()

Thread 18 (Thread 1080400224 (LWP 21541)):

#0 0x0000003158608baf in pthread_cond_timedwait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2ba5 in os::Linux::safe_cond_timedwait ()

#2 0x0000002a95acace4 in Monitor::wait ()

#3 0x0000002a95be1ab2 in VMThread::loop ()

#4 0x0000002a95be173a in VMThread::run ()

#5 0x0000002a95ae3615 in _start ()

#6 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#7 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#8 0x0000000000000000 in ? ()

Thread 17 (Thread 1081452896 (LWP 21542)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95ad91e5 in ObjectMonitor::wait ()

#3 0x0000002a95b73ad4 in ObjectSynchronizer::wait ()

#4 0x0000002a959609bf in JVM_MonitorWait ()

#5 0x0000002a9940e562 in ? ()

#6 0x0000000040759ac0 in ? ()

#7 0x0000002aa193de10 in ? ()

#8 0x0000000040759a00 in ? ()

#9 0x00000000407599e8 in ? ()

#10 0x0000000000000000 in ? ()

Thread 16 (Thread 1082505568 (LWP 21543)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95ad91e5 in ObjectMonitor::wait ()

#3 0x0000002a95b73ad4 in ObjectSynchronizer::wait ()

#4 0x0000002a959609bf in JVM_MonitorWait ()

#5 0x0000002a9940e562 in ? ()

#6 0x000000004085aa30 in ? ()

#7 0x0000002ae1d57100 in ? ()

#8 0x000000004085a990 in ? ()

#9 0x0000002a958ec243 in InterpreterRuntime::monitorenter ()

#10 0x0000002a99404efe in ? ()

#11 0x0000000000000000 in ? ()

Thread 15 (Thread 1083558240 (LWP 21544)):

#0 0x000000315860a66f in sem_wait () from /lib64/tls/libpthread.so.0

#1 0x0000002a95ae39f8 in check_pending_signals ()

#2 0x0000002a95ade495 in signal_thread_entry ()

#3 0x0000002a95ba0c4e in JavaThread::run ()

#4 0x0000002a95ae3615 in _start ()

#5 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#6 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#7 0x0000000000000000 in ? ()

Thread 14 (Thread 1084610912 (LWP 21545)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95acadb4 in Monitor::wait ()

#3 0x0000002a95839f21 in CompileQueue::get ()

#4 0x0000002a9583bf61 in CompileBroker::compiler_thread_loop ()

#5 0x0000002a95ba6499 in compiler_thread_entry ()

#6 0x0000002a95ba0c4e in JavaThread::run ()

#7 0x0000002a95ae3615 in _start ()

#8 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#9 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#10 0x0000000000000000 in ? ()

Thread 13 (Thread 1085663584 (LWP 21546)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95acadb4 in Monitor::wait ()

#3 0x0000002a95839f21 in CompileQueue::get ()

#4 0x0000002a9583bf61 in CompileBroker::compiler_thread_loop ()

#5 0x0000002a95ba6499 in compiler_thread_entry ()

#6 0x0000002a95ba0c4e in JavaThread::run ()

#7 0x0000002a95ae3615 in _start ()

#8 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#9 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#10 0x0000000000000000 in ? ()

Thread 12 (Thread 1086716256 (LWP 21547)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95acadb4 in Monitor::wait ()

#3 0x0000002a95839f21 in CompileQueue::get ()

#4 0x0000002a9583bf61 in CompileBroker::compiler_thread_loop ()

#5 0x0000002a95ba6499 in compiler_thread_entry ()

#6 0x0000002a95ba0c4e in JavaThread::run ()

#7 0x0000002a95ae3615 in _start ()

#8 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#9 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#10 0x0000000000000000 in ? ()

Thread 11 (Thread 1087768928 (LWP 21548)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95acac39 in Monitor::wait ()

#3 0x0000002a95aa0c2f in LowMemoryDetector::low_memory_detector_thread_entry

#4 0x0000002a95ba0c4e in JavaThread::run ()

#5 0x0000002a95ae3615 in _start ()

#6 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#7 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#8 0x0000000000000000 in ? ()

Thread 10 (Thread 1088821600 (LWP 21549)):

#0 0x0000003158608baf in pthread_cond_timedwait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae1b46 in os::sleep ()

#2 0x0000002a95b9fede in WatcherThread::run ()

#3 0x0000002a95ae3615 in _start ()

#4 0x000000315860610a in start_thread () from /lib64/tls/libpthread.so.0

#5 0x0000003157bc68c3 in clone () from /lib64/tls/libc.so.6

#6 0x0000000000000000 in ? ()

Thread 9 (Thread 1089874272 (LWP 21550)):

#0 0x0000003157bbd9a2 in poll () from /lib64/tls/libc.so.6

#1 0x0000002ae7add6de in Java_sun_nio_ch_PollArrayWrapper_interrupt ()

#2 0x0000002ae7add5f1 in Java_sun_nio_ch_PollArrayWrapper_poll0 ()

#3 0x0000002a9973849b in ? ()

#4 0x0000002aa195e8e8 in ? ()

#5 0x0000000000000000 in ? ()

Thread 8 (Thread 1090926944 (LWP 21551)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95acb253 in Parker::park ()

#3 0x0000002a95bc821e in Unsafe_Park ()

#4 0x0000002a9940e562 in ? ()

#5 0x0000002a9c510d78 in ? ()

#6 0x0000000041062880 in ? ()

#7 0x00000000410628c8 in ? ()

#8 0x0000002a99404d36 in ? ()

#9 0x0000000041062870 in ? ()

#10 0x0000000000000000 in ? ()

Thread 7 (Thread 1091979616 (LWP 21552)):

#0 0x0000003158608baf in pthread_cond_timedwait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2bdf in os::Linux::safe_cond_timedwait ()

#2 0x0000002a95ad917f in ObjectMonitor::wait ()

#3 0x0000002a95b73ad4 in ObjectSynchronizer::wait ()

#4 0x0000002a959609bf in JVM_MonitorWait ()

#5 0x0000002a997887bf in ? ()

#6 0x0000002aa195dc40 in ? ()

#7 0x0000000041163a28 in ? ()

#8 0x0000002aa195dc40 in ? ()

#9 0x0000002aa195dc40 in ? ()

#10 0x0000002ae5222c00 in ? ()

#11 0x0000002ae5222c00 in ? ()

#12 0x0000000041163a28 in ? ()

#13 0x0000002a997c3868 in ? ()

#14 0x0000002ae5222c00 in ? ()

#15 0x0000000041163a28 in ? ()

#16 0x0000002a9ca2508c in ? ()

#17 0x0000000041163a58 in ? ()

#18 0x0000002aa195dc40 in ? ()

#19 0x0000002aa195dcd0 in ? ()

#20 0x0000002ae5222c00 in ? ()

#21 0x0000002a9c56a8a8 in ? ()

#22 0x0000002aa195dc40 in ? ()

#23 0x0000000000000000 in ? ()

Thread 6 (Thread 1093032288 (LWP 21553)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95acb253 in Parker::park ()

#3 0x0000002a95bc821e in Unsafe_Park ()

#4 0x0000002a9940e562 in ? ()

#5 0x0000000000000001 in ? ()

#6 0x0000000000000000 in ? ()

Thread 5 (Thread 1094084960 (LWP 21556)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95ad91e5 in ObjectMonitor::wait ()

#3 0x0000002a95b73ad4 in ObjectSynchronizer::wait ()

#4 0x0000002a959609bf in JVM_MonitorWait ()

#5 0x0000002a9940e562 in ? ()

#6 0x0000000041365c30 in ? ()

#7 0x0000002ae4d9a3c8 in ? ()

#8 0x0000002ae7850500 in ? ()

#9 0x0000000041365bf8 in ? ()

#10 0x0000000000000000 in ? ()

Thread 4 (Thread 1095137632 (LWP 21557)):

#0 0x000000315860b3df in __accept_nocancel () from /lib64/tls/libpthread.so.0

#1 0x0000002ae791035f in NET_Accept ()

#2 0x0000002ae790c71f in Java_java_net_PlainSocketImpl_socketAccept ()

#3 0x0000002a9940e562 in ? ()

#4 0x0000002ae89f8aa0 in ? ()

#5 0x0000002acd7f3fd8 in ? ()

#6 0x0000002ae89f8aa0 in ? ()

#7 0x0000000041466aa8 in ? ()

#8 0x0000002a9ca6f4f0 in ? ()

#9 0x0000000041466b08 in ? ()

#10 0x0000002a9ca73108 in ? ()

#11 0x0000000000000000 in ? ()

Thread 3 (Thread 1096190304 (LWP 21558)):

#0 0x0000003158608baf in pthread_cond_timedwait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2bdf in os::Linux::safe_cond_timedwait ()

#2 0x0000002a95ae1990 in os::sleep ()

#3 0x0000002a95972094 in JVM_Sleep ()

#4 0x0000002a997a257c in ? ()

#5 0x0000002a9c513460 in ? ()

#6 0x0000002aa1960870 in ? ()

#7 0x0000002ab2ddded0 in ? ()

#8 0x0000002ab2ddf118 in ? ()

#9 0x0000002ae4e7a320 in ? ()

#10 0x0000002ae4e7a320 in ? ()

#11 0x0000000000000001 in ? ()

#12 0x0000002a9978ae6c in ? ()

#13 0x0000000000000000 in ? ()

Thread 2 (Thread 1097242976 (LWP 21559)):

#0 0x00000031586089ba in pthread_cond_wait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2a8a in os::Linux::safe_cond_wait ()

#2 0x0000002a95ad91e5 in ObjectMonitor::wait ()

#3 0x0000002a95b73ad4 in ObjectSynchronizer::wait ()

#4 0x0000002a959609bf in JVM_MonitorWait ()

#5 0x0000002a9940e562 in ? ()

#6 0x0000000041668aa0 in ? ()

#7 0x0000002a95ad84ed in ObjectMonitor::enter ()

#8 0x0000002a99404efe in ? ()

#9 0x0000000000000000 in ? ()

Thread 1 (Thread 182894172512 (LWP 21536)):

#0 0x0000003158608baf in pthread_cond_timedwait@@GLIBC_2.3.2 ()

#1 0x0000002a95ae2bdf in os::Linux::safe_cond_timedwait ()

#2 0x0000002a95ae1990 in os::sleep ()

#3 0x0000002a95972094 in JVM_Sleep ()

#4 0x0000002a997a257c in ? ()

#5 0x0000002a9c513460 in ? ()

#6 0x0000002aa1901698 in ? ()

#7 0x00000000ffffffff in ? ()

#8 0x0000002aa19013f0 in ? ()

#9 0x0000000040117fb0 in ? ()

#10 0x0000002ab2dddbf8 in ? ()

#11 0x0000002aa1901698 in ? ()

#12 0x0000002a997a2028 in ? ()

#13 0x0000002aa19013f0 in ? ()

#14 0x0000000040117fb0 in ? ()

#15 0x0000002aa1901698 in ? ()

#16 0x0000002a99716f38 in ? ()

#17 0x0000002aa19016a8 in ? ()

#18 0x0000002a00000001 in ? ()

#19 0x0000002add7fb1b8 in ? ()

#20 0x0000000040117fb0 in ? ()

#21 0x0000002add801b00 in ? ()

#22 0x0000002aa1901698 in ? ()

#23 0x0000002add7fd318 in ? ()

#24 0x0000007fbfffc048 in ? ()

#25 0x0000002a9c9b87a8 in ? ()

#26 0x0000007fbfffc0e0 in ? ()

#27 0x0000002ab3273bf0 in ? ()

#28 0x00000044ffffffff in ? ()

#29 0x0000002aa1960870 in ? ()

#30 0x0000002add802b98 in ? ()

#31 0x0000002add802f90 in ? ()

#32 0x0000000040117fb0 in ? ()

#33 0x0000000000000003 in ? ()

#34 0x0000002a9944ff60 in ? ()

#35 0x00000000000000f9 in ? ()

#36 0x0000002a994162b3 in ? ()

#37 0x0000007fbfffc010 in ? ()

#38 0x0000002a9c9b87a8 in ? ()

#39 0x0000007fbfffc0e0 in ? ()

#40 0x0000002a9c9bc2c0 in ? ()

#41 0x0000002a9cdaee10 in ? ()

#42 0x0000002a9c9b8ba0 in ? ()

#43 0x0000007fbfffc0d8 in ? ()

#44 0x0000007fbfffc120 in ? ()

#45 0x0000002a99404efe in ? ()

#46 0x0000000000000000 in ? ()

lockingAndBlockinga at 2007-7-14 19:25:23 >