JVM Hotspot compiler crashed
Hello,
The JVM crashed in one of our Solaris production environments. The cause of the crash appears to be the HotSpot compiler. It seems that the JVM was trying to compile frequently used code and then it crashed. I have checked the forums and I found one other message with the same Internal Error ID. However, in that message, it is suggested that the JVM itself is running out of C malloc space (http://forum.java.sun.com/thread.jspa?forumID=37&threadID=5127551).
I have verified that the JVM was not running out of memory when it crashed. There was over 9 GB of swap and physical memory available to the JVM when the crash happened.
Below are the contents of the hs_err_pid file.
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# Internal Error (434F44452255464645520E4350500060 01), pid=2355, tid=13
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
T H R E A D
Current thread (0x0017fcb8): JavaThread "CompilerThread0" daemon [_thread_in_native, id=13]
Stack: [0xa2300000,0xa2380000), sp=0xa237e370, free space=504k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6f5644]
V [libjvm.so+0x4369b0]
V [libjvm.so+0x1b5140]
V [libjvm.so+0x21b8b0]
V [libjvm.so+0x227b70]
V [libjvm.so+0x224ce4]
V [libjvm.so+0x21ab78]
V [libjvm.so+0x282d78]
V [libjvm.so+0x278d04]
V [libjvm.so+0x2799c0]
V [libjvm.so+0x336548]
V [libjvm.so+0x2deae8]
V [libjvm.so+0x6691b0]
Current CompileTask:
opto:4113gov.nist.javax.sip.stack.SIPDialog.setLastResponse(Lgov/nist/javax/sip/stack/SIPTransaction;Lgov/nist/javax/sip/message/SIPResponse;)V (954 bytes)
P R O C E S S
Java Threads: ( => current thread )
0x01a70548 JavaThread "RMI ConnectionExpiration-[90.0.64.65:49888]" daemon [_thread_blocked, id=121153]
0x01d8a5e0 JavaThread "RMI RenewClean-[90.0.64.65:49888]" daemon [_thread_blocked, id=5835]
0x005433e0 JavaThread "Thread-29" [_thread_in_native, id=1227]
0x0110a820 JavaThread "Thread-30" [_thread_blocked, id=1226]
0x016d2ae8 JavaThread "RMI RenewClean-[90.0.64.64:32827]" daemon [_thread_blocked, id=167]
0x00d31a28 JavaThread "UDPMessageChannelThread" daemon [_thread_blocked, id=120]
0x0151a1a8 JavaThread "UDPMessageProcessorThread" daemon [_thread_in_native, id=119]
0x01519f38 JavaThread "MitelTCPServer" [_thread_in_native, id=118]
0x00f1ced8 JavaThread "MitelTCPHandler" [_thread_in_native, id=117]
0x00f1d478 JavaThread "MinetEncryptedServer" [_thread_blocked, id=116]
0x00f1afb8 JavaThread "JainMgcpStackImpl" [_thread_in_native, id=115]
0x0079baf0 JavaThread "Thread-24" daemon [_thread_blocked, id=113]
0x00d22018 JavaThread "Thread-20" daemon [_thread_blocked, id=111]
0x00d21e50 JavaThread "Thread-18" daemon [_thread_blocked, id=110]
0x008e4090 JavaThread "Thread-15" [_thread_in_native, id=73]
0x01069af8 JavaThread "Thread-16" [_thread_blocked, id=72]
0x010e5720 JavaThread "Thread-14" [_thread_blocked, id=69]
0x000388d8 JavaThread "DestroyJavaVM" [_thread_blocked, id=1]
0x01e6b3b0 JavaThread "RTSEngine Watchdog" [_thread_blocked, id=68]
0x007826e0 JavaThread "RTSEngine" [_thread_in_Java, id=67]
0x00be1cd8 JavaThread "DBEngine Watchdog" [_thread_blocked, id=66]
0x00bdde38 JavaThread "DBEngine" [_thread_blocked, id=65]
0x00fc8a08 JavaThread "DefaultDomain:class=SnmpV3AdaptorServer,protocol=snmp,port=8163" [_thread_in_native, id=63]
0x00605e60 JavaThread "DefaultDomain:class=HtmlAdaptorServer,protocol=html,port=9001" [_thread_in_native, id=62]
0x001f98a0 JavaThread "CDR Processor" [_thread_blocked, id=61]
0x00fcb680 JavaThread "Python Server" [_thread_in_native, id=60]
0x00a15bc0 JavaThread "CDR File Manager" [_thread_blocked, id=59]
0x00542bb0 JavaThread "MemoryLogger" [_thread_blocked, id=58]
0x00541d20 JavaThread "Globals Executioner" [_thread_blocked, id=57]
0x00a16068 JavaThread "Remote Command Executioner" [_thread_blocked, id=56]
0x0157ab78 JavaThread "CallpAuditor" [_thread_blocked, id=55]
0x00664068 JavaThread "VACommClient" [_thread_blocked, id=54]
0x003f34e0 JavaThread "Thread-11" [_thread_blocked, id=50]
0x005844a8 JavaThread "Thread-10" daemon [_thread_blocked, id=34]
0x00f95e60 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=32]
0x010821f0 JavaThread "GC Daemon" daemon [_thread_blocked, id=30]
0x00587928 JavaThread "RMI Reaper" [_thread_blocked, id=29]
0x015fd208 JavaThread "Timer-3" [_thread_blocked, id=28]
0x009b4200 JavaThread "EventScannerThread" [_thread_blocked, id=27]
0x006df418 JavaThread "Timer-2" [_thread_blocked, id=26]
0x0051d958 JavaThread "MemoryHandlerEx" [_thread_blocked, id=20]
0x0035cef8 JavaThread "RMI TCP Accept-4003" daemon [_thread_in_native, id=18]
0x00357d70 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=17]
0x003555c8 JavaThread "Timer-0" daemon [_thread_blocked, id=16]
0x00181d90 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
0x00180c10 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14]
=>0x0017fcb8 JavaThread "CompilerThread0" daemon [_thread_in_native, id=13]
0x0017e370 JavaThread "AdapterThread" daemon [_thread_blocked, id=12]
0x0017d1e8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11]
0x0017c118 JavaThread "JDWP Event Helper Thread" daemon [_thread_blocked, id=10]
0x00174030 JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_in_native, id=9]
0x00171420 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=8]
0x00164aa8 JavaThread "Finalizer" daemon [_thread_blocked, id=7]
0x00164568 JavaThread "Reference Handler" daemon [_thread_blocked, id=6]
Other Threads:
0x00162488 VMThread [id=5]
0x003b28f0 WatcherThread [id=19]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
par new generationtotal 7424K, used 3798K [0xa5800000, 0xa6000000, 0xa6000000)
eden space 6656K, 45% used [0xa5800000, 0xa5af5b48, 0xa5e80000)
from space 768K, 100% used [0xa5e80000, 0xa5f40000, 0xa5f40000)
tospace 768K,0% used [0xa5f40000, 0xa5f40000, 0xa6000000)
concurrent mark-sweep generation total 1220608K, used 354273K [0xa6000000, 0xf0800000, 0xf0800000)
concurrent-mark-sweep perm gen total 46400K, used 27804K [0xf0800000, 0xf3550000, 0xf8800000)
Dynamic libraries:
0x00010000 /usr/jdk1.5.0_07/bin/java
0xff370000 /usr/lib/libthread.so.1
0xff3fa000 /usr/lib/libdl.so.1
0xff280000 /usr/lib/libc.so.1
0xff3a0000 /usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
0xfe800000 /usr/jdk1.5.0_07/jre/lib/sparc/server/libjvm.so
0xff240000 /usr/lib/libsocket.so.1
0xff220000 /usr/lib/libsched.so.1
0xff1f0000 /usr/lib/libCrun.so.1
0xff1a0000 /usr/lib/libm.so.1
0xff080000 /usr/lib/libnsl.so.1
0xff170000 /usr/lib/libmp.so.2
0xff050000 /usr/jdk1.5.0_07/jre/lib/sparc/native_threads/libhpi.so
0xfe7b0000 /usr/jdk1.5.0_07/jre/lib/sparc/libjdwp.so
0xfe770000 /usr/jdk1.5.0_07/jre/lib/sparc/libverify.so
0xfe720000 /usr/jdk1.5.0_07/jre/lib/sparc/libjava.so
0xfe700000 /usr/jdk1.5.0_07/jre/lib/sparc/libzip.so
0xfbfb0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2
0xfbf90000 /usr/jdk1.5.0_07/jre/lib/sparc/libdt_socket.so
0xfbee0000 /usr/lib/nss_files.so.1
0xfbec0000 /usr/jdk1.5.0_07/jre/lib/sparc/libmanagement.so
0xfbdd0000 /usr/jdk1.5.0_07/jre/lib/sparc/libnet.so
0xf8ab0000 /usr/jdk1.5.0_07/jre/lib/sparc/libnio.so
0xf8a90000 /usr/lib/librt.so.1
0xf8860000 /usr/lib/libaio.so.1
0xf8840000 /usr/lib/libmd5.so.1
0xf8820000 /usr/lib/libsendfile.so.1
0xa57e0000 /usr/jdk1.5.0_07/jre/lib/sparc/librmi.so
0x9dc00000 /nci/vks.2.0.9.0.1/callp/libmgcpca.so.3.0.0.0.0
0xa5690000 /usr/local/lib/libstdc++.so.2.10.0
VM Arguments:
jvm_args: -XX:+UseISM -Xms1200m -Xmx1200m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxPermSize=128m -XX:SurvivorRatio=8 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:-TraceClassUnloading -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=4001 -Dcom.sun.management.jmxremote.port=4003 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/nci/vks/callp/jmxremote.password -Dnci.callp.state=2
java_command: com.nci.callp.MainApplication
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java
CLASSPATH=:/usr/java/lib/tools.jar:/usr/java/jre/lib/rt.jar:.:/nci/vks/lib/bitlib.jar:/nci/vks/lib/nist.jar:/nci/vks/lib/antlrall.jar:/nci/vks/lib/common.jar:/nci/vks/lib/jconn2.jar:/nci/vks/callp/callp.jar:/nci/vks/callp/lib/3comlib.jar:/nci/vks/callp/lib/jcert.jar:/nci/vks/callp/lib/jnet.jar:/nci/vks/callp/lib/jsse.jar:/nci/vks/callp/lib/mitellib.jar:/nci/vks/mediaserver/mediaserver.jar:/nci/vks/snmp/snmp.jar:/nci/vks/snmp/lib/solaris/jdmkrt.jar:/nci/vks/snmp/lib/solaris/jdmktk.jar:/nci/vks/snmp/lib/solaris/jsnmpapi.jar:/nci/vks/auditor/auditor.jar:.:/nci/vks/lib/bitlib.jar:/nci/vks/lib/nist.jar:/nci/vks/lib/antlrall.jar:/nci/vks/lib/common.jar:/nci/vks/lib/jconn2.jar:/vks_shared/callp/resources:/nci/vks/callp/callp.jar:/nci/vks/callp/lib/3comlib.jar:/nci/vks/callp/lib/jcert.jar:/nci/vks/callp/lib/jnet.jar:/nci/vks/callp/lib/jsse.jar:/nci/vks/callp/lib/mitellib.jar:/nci/vks/callp/lib/cpl.jar:/nci/vks/callp/lib/asbsdk.jar:/nci/vks/mediaserver/mediaserver.jar:/nci/vks/snmp/snmp.jar:/nci/vks/snmp/lib/solaris/jdmkrt.jar:/nci/vks/snmp/lib/solaris/jdmktk.jar:/nci/vks/snmp/lib/solaris/jsnmpapi.jar:/nci/vks/lib/commons-collections-3.0.jar:/nci/vks/lib/commons-lang-2.0.jar:/nci/vks/lib/jython.jar:/nci/vks/lib/castor-0.9.7.jar:/usr/java/lib/tools.jar:/nci/vks/lib/commons-logging-1.0.3.jar://nci/vks/lib/common.nist.jar://nci/vks/lib/concurrent.jar://nci/vks/lib/log4j-1.2.8.jar
PATH=/nci/sybase/SYBSsa9/bin32:/nci/sybase/shared9/sybcentral43:/usr/sbin:/usr/bin:/nci/sybase/SYBSsa9/bin:/usr/sbin:/opt/SUNWcgha/sbin:/nci/vks/bin:/usr/local/bin:/usr/ccs/bin:/opt/SUNWcgha/sbin:/nci/mysql/bin
LD_LIBRARY_PATH=/usr/jdk1.5.0_07/jre/lib/sparc/server:/usr/jdk1.5.0_07/jre/lib/sparc:/usr/jdk1.5.0_07/jre/../lib/sparc:/nci/sybase/SYBSsa9/lib32:/nci/sybase/SYBSsa9/lib64:/nci/sybase/SYBSsa9/jre142/lib/sparc/client:/nci/sybase/SYBSsa9/jre142/lib/sparc:/nci/sybase/SYBSsa9/jre142/lib/sparc/native_threads:/nci/sybase/SYBSsa9/drivers/lib::/usr/lib:/nci/sybase/SYBSsa9/lib:/usr/local/lib
SHELL=/usr/bin/bash
HOSTTYPE=sparc
OSTYPE=solaris2.9
MACHTYPE=sparc-sun-solaris2.9
Signal Handlers:
SIGSEGV: [libjvm.so+0x6f60b8], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x6f60b8], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x276398], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x276398], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x276398], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0x66b714], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0x276398], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0x66a38c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x66a38c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x66a38c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
S Y S T E M
OS:Solaris 9 9/04 s9s_u7wos_09 SPARC
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 29 June 2004
uname:SunOS 5.9 Generic_118558-28 sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:0.49 0.59 0.61
CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 4194304k(839536k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_07-b03) for solaris-sparc, built on May 3 2006 01:22:35 by unknown with unknown Workshop:0x550
