Bizzar JVM crash on dual Xeon EMT64 machine
I am posting this one here to see if anyone else has the same or similar experience. I have an Java application running on a Windows Server 2003 SP1 OS with dual Xeon EMT64. It is running on Sun's JDK 1.5 Update 3, and every couple of hours the JVM instance just crash with the following error log:
######################################################
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x77bbc4ba, pid=2716, tid=2776
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# C [MSVCRT.dll+0x1c4ba]
#
T H R E A D
Current thread (0x00a93ca0): JavaThread "FileStore" daemon [_thread_in_vm, id=2776]
siginfo: ExceptionCode=0xc0000005, reading address 0x64b8c063
Registers:
EAX=0x49b8ae28, EBX=0x009d1ea4, ECX=0x49b8bf8c, EDX=0x64b8c063
ESP=0x4854f8d0, EBP=0x4854f8f0, ESI=0x000fffff, EDI=0x0000001b
EIP=0x77bbc4ba, EFLAGS=0x00010286
Top of Stack: (sp=0x4854f8d0)
0x4854f8d0:000000ba 000000ba 7c82f9fd 77bf3a7c
0x4854f8e0:000000d0 49b8bf8c ffffffff 00000008
0x4854f8f0:4854f924 77bbcf2e 009d1ea4 000000ba
0x4854f900:00000000 7c82f9fd 00000000 4854f8fc
0x4854f910:4854f4fc 4854fdf4 77bc6c74 77ba2100
0x4854f920:00000000 4854f93c 77bbd06d 000000ba
0x4854f930:000000ba 0000005c 000000ba 4854f974
0x4854f940:6da52456 000000ba 00a93ca0 000000ba
Instructions: (pc=0x77bbc4ba)
0x77bbc4aa:00 00 d1 e1 47 85 c9 7d f9 8b 4d f4 8b 54 f9 04
0x77bbc4ba:8b 0a 2b 4d f0 8b f1 c1 fe 04 4e 83 fe 3f 89 4d
Stack: [0x48450000,0x48550000), sp=0x4854f8d0, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [MSVCRT.dll+0x1c4ba]
C [MSVCRT.dll+0x1cf2e]
C [MSVCRT.dll+0x1d06d]
V [jvm.dll+0x212456]
C [java.dll+0x8ac2]
j java.io.WinNTFileSystem.list(Ljava/io/File;)[Ljava/lang/String;+0
j java.io.File.list()[Ljava/lang/String;+20
j org.apache.catalina.session.FileStore.keys()[Ljava/lang/String;+15
j org.apache.catalina.session.StoreBase.processExpires()V+15
j org.apache.catalina.session.StoreBase.run()V+12
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [jvm.dll+0xefce0]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.io.WinNTFileSystem.list(Ljava/io/File;)[Ljava/lang/String;+0
j java.io.File.list()[Ljava/lang/String;+20
j org.apache.catalina.session.FileStore.keys()[Ljava/lang/String;+15
j org.apache.catalina.session.StoreBase.processExpires()V+15
j org.apache.catalina.session.StoreBase.run()V+12
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
P R O C E S S
Java Threads: ( => current thread )
0x51dbaa20 JavaThread "LISTENER" [_thread_in_native, id=1656]
0x51dbae80 JavaThread "Thread-69" [_thread_blocked, id=1436]
0x51db1270 JavaThread "GenericSocketServerFE" [_thread_in_native, id=2024]
0x51db2a30 JavaThread "MainSocketServerFE" [_thread_in_native, id=748]
0x51db2800 JavaThread "Thread-53" [_thread_blocked, id=2368]
0x51da4090 JavaThread "statuspoll-1" [_thread_blocked, id=2124]
0x51da4210 JavaThread "statuspoll" [_thread_blocked, id=1156]
0x51dad200 JavaThread "datapoll-20" [_thread_blocked, id=2180]
0x51dad380 JavaThread "datapoll-19" [_thread_blocked, id=772]
0x51dade80 JavaThread "datapoll-18" [_thread_blocked, id=2600]
0x51daca80 JavaThread "datapoll-17" [_thread_blocked, id=1304]
0x51dab680 JavaThread "datapoll-16" [_thread_blocked, id=480]
0x51daa260 JavaThread "datapoll-15" [_thread_blocked, id=1332]
0x51daade0 JavaThread "datapoll-14" [_thread_blocked, id=972]
0x51da9a00 JavaThread "datapoll-13" [_thread_blocked, id=2664]
0x51da87f0 JavaThread "datapoll-12" [_thread_blocked, id=1940]
0x51da74f0 JavaThread "datapoll-11" [_thread_blocked, id=2528]
0x51da60a0 JavaThread "datapoll-10" [_thread_blocked, id=2108]
0x51da6d40 JavaThread "datapoll-9" [_thread_blocked, id=2104]
0x51da5a00 JavaThread "datapoll-8" [_thread_blocked, id=4036]
0x51da4d30 JavaThread "datapoll-7" [_thread_blocked, id=528]
0x51da3900 JavaThread "datapoll-6" [_thread_blocked, id=1460]
0x51da2460 JavaThread "datapoll-5" [_thread_blocked, id=1424]
0x51da2e80 JavaThread "datapoll-4" [_thread_blocked, id=1340]
0x51da1a80 JavaThread "datapoll-3" [_thread_blocked, id=1136]
0x51da0780 JavaThread "datapoll-2" [_thread_blocked, id=1620]
0x51d8d300 JavaThread "datapoll-1" [_thread_blocked, id=744]
0x51d8ce80 JavaThread "datapoll" [_thread_blocked, id=1100]
0x00adf210 JavaThread "Ajp13Processor[8009][2]" daemon [_thread_blocked, id=2076]
0x00adcc10 JavaThread "Ajp13Processor[8009][1]" daemon [_thread_in_native, id=2720]
0x00ad9490 JavaThread "Ajp13Processor[8009][0]" daemon [_thread_blocked, id=2724]
0x00addb20 JavaThread "Ajp13Connector[8009]" daemon [_thread_in_native, id=2728]
0x00a8b100 JavaThread "HostConfig[localhost]" daemon [_thread_blocked, id=2732]
0x00a8ce80 JavaThread "JspRuntimeContext[\HIPATH~1]" daemon [_thread_blocked, id=2744]
0x00a94ae0 JavaThread "StandardManager[]" daemon [_thread_blocked, id=2772]
=>0x00a93ca0 JavaThread "FileStore" daemon [_thread_in_vm, id=2776]
0x51d95e80 JavaThread "TopoTreeCmdExecutorThread" [_thread_blocked, id=2800]
0x51d87d70 JavaThread "controller-monitor-thread" [_thread_blocked, id=4020]
0x51d87bf0 JavaThread "Thread-50" [_thread_blocked, id=2280]
0x51d82340 JavaThread "Thread-49" [_thread_blocked, id=2480]
0x51d825c0 JavaThread "TRAPINPUT" [_thread_blocked, id=1900]
0x51d817c0 JavaThread "TRAPSTORE0" [_thread_in_native, id=1904]
0x51d829f0 JavaThread "COMMONTRAPNOTIFIER" [_thread_blocked, id=364]
0x51d7c8a0 JavaThread "Thread-48" [_thread_blocked, id=356]
0x51d747e0 JavaThread "SUMMainServerThread" [_thread_in_native, id=140]
0x51d73c80 JavaThread "1-1" [_thread_blocked, id=2044]
0x00ad8da0 JavaThread "1" [_thread_blocked, id=2008]
0x00ad5b80 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=3956]
0x00aca900 JavaThread "policy-10" [_thread_blocked, id=3904]
0x00ac90e0 JavaThread "policy-9" [_thread_blocked, id=2140]
0x00ac9ca0 JavaThread "policy-8" [_thread_blocked, id=2524]
0x00ac8880 JavaThread "policy-7" [_thread_blocked, id=3776]
0x00aa4020 JavaThread "policy-6" [_thread_blocked, id=2612]
0x00a75070 JavaThread "policy-5" [_thread_blocked, id=1296]
0x00ac7a10 JavaThread "policy-4" [_thread_blocked, id=3748]
0x00ac6150 JavaThread "policy-3" [_thread_blocked, id=3912]
0x00ac5bd0 JavaThread "policy-2" [_thread_blocked, id=1928]
0x00ac48a0 JavaThread "policy-1" [_thread_blocked, id=3732]
0x00ac2140 JavaThread "policy" [_thread_blocked, id=980]
0x00abeb70 JavaThread "RMI RenewClean-[10.1.99.136:1129]" daemon [_thread_blocked, id=2576]
0x00ab8b70 JavaThread "RMI TCP Accept-1099" daemon [_thread_in_native, id=292]
0x00a983d0 JavaThread "Thread-23" [_thread_blocked, id=2608]
0x00a981a0 JavaThread "Thread-44" [_thread_blocked, id=1992]
0x00a98e80 JavaThread "Thread-43" [_thread_in_native, id=1028]
0x00a866d0 JavaThread "cmd /c ""C:\PROGRA~1\Siemens\HIPATH~1\apache\bin\apache" -d "C:\PROGRA~1\Siemens\HIPATH~1\apache"" -f conf/httpd.conf " [_thread_in_native, id=2676]
0x00a75c60 JavaThread "BEModuleDispatcher-2" [_thread_blocked, id=3876]
0x00a741c0 JavaThread "BEModuleDispatcher-1" [_thread_blocked, id=1948]
0x00a74520 JavaThread "Thread-5" [_thread_in_native, id=1960]
0x00a74a10 JavaThread "BEModuleDispatcher" [_thread_blocked, id=2052]
0x00a388b0 JavaThread "KeepAliveMonitor" [_thread_blocked, id=2620]
0x00a2d9e0 JavaThread "main-10" [_thread_blocked, id=3932]
0x00a2c5e0 JavaThread "main-9" [_thread_blocked, id=520]
0x00a2b320 JavaThread "main-8" [_thread_blocked, id=544]
0x00a2a080 JavaThread "main-7" [_thread_blocked, id=2320]
0x00a291a0 JavaThread "main-6" [_thread_blocked, id=472]
0x00a28020 JavaThread "main-5" [_thread_blocked, id=2640]
0x00a1f110 JavaThread "main-4" [_thread_blocked, id=3936]
0x00a12a80 JavaThread "main-3" [_thread_blocked, id=1336]
0x009f9e80 JavaThread "SchedulerForFailOver-2" [_thread_blocked, id=2092]
0x00a260f0 JavaThread "main-2" [_thread_blocked, id=504]
0x00a26710 JavaThread "SchedulerForFailOver-1" [_thread_blocked, id=2236]
0x00a25600 JavaThread "main-1" [_thread_blocked, id=1504]
0x00a23de0 JavaThread "SchedulerForFailOver" [_thread_blocked, id=1220]
0x00a24e50 JavaThread "main" [_thread_blocked, id=3736]
0x00a15730 JavaThread "Thread-4" [_thread_in_native, id=784]
0x00a15da0 JavaThread "Thread-3" [_thread_blocked, id=2148]
0x00a0b540 JavaThread "GC Daemon" daemon [_thread_blocked, id=1520]
0x00a0a2f0 JavaThread "RMI Reaper" [_thread_blocked, id=2160]
0x00a091f0 JavaThread "Timer-0" daemon [_thread_blocked, id=1168]
0x00a094f0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=2156]
0x009f67d0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1796]
0x009f5b20 JavaThread "CompilerThread1" daemon [_thread_blocked, id=1072]
0x009f4a00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=2464]
0x009f3910 JavaThread "AdapterThread" daemon [_thread_blocked, id=2588]
0x009f27d0 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3568]
0x009f1b20 JavaThread "Finalizer" daemon [_thread_blocked, id=2592]
0x009f0880 JavaThread "Reference Handler" daemon [_thread_blocked, id=2688]
0x009e24e0 JavaThread "main" [_thread_blocked, id=2712]
Other Threads:
0x009efa40 VMThread [id=2692]
0x009f72e0 WatcherThread [id=3824]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGentotal 27840K, used 1576K [0x3fb90000, 0x41d00000, 0x46d50000)
eden space 23744K, 6% used [0x3fb90000,0x3fd1a1b8,0x412c0000)
from space 4096K, 0% used [0x41780000,0x41780000,0x41b80000)
tospace 4864K, 0% used [0x412c0000,0x412c0000,0x41780000)
PSOldGentotal 15104K, used 14054K [0x06d50000, 0x07c10000, 0x3fb90000)
object space 15104K, 93% used [0x06d50000,0x07b09ba8,0x07c10000)
PSPermGentotal 23296K, used 20545K [0x02d50000, 0x04410000, 0x06d50000)
object space 23296K, 88% used [0x02d50000,0x041607b0,0x04410000)
Dynamic libraries:
0x00400000 - 0x00417000 C:\Program Files\Siemens\HiPath Wireless Manager\lib\JavaService.exe
0x7c800000 - 0x7c8c0000 C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f42000 C:\WINDOWS\system32\kernel32.dll
0x77f50000 - 0x77fec000 C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cef000 C:\WINDOWS\system32\RPCRT4.dll
0x6d840000 - 0x6dbcb000 C:\PROGRA~1\Siemens\HIPATH~1\langley\jre\bin\server\jvm.dll
0x77380000 - 0x77412000 C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c48000 C:\WINDOWS\system32\GDI32.dll
0x76aa0000 - 0x76acd000 C:\WINDOWS\system32\WINMM.dll
0x77ba0000 - 0x77bfa000 C:\WINDOWS\system32\MSVCRT.dll
0x71bc0000 - 0x71bc8000 C:\WINDOWS\system32\rdpsnd.dll
0x771f0000 - 0x77201000 C:\WINDOWS\system32\WINSTA.dll
0x71c40000 - 0x71c98000 C:\WINDOWS\system32\NETAPI32.dll
0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL
0x6d2f0000 - 0x6d2f8000 C:\PROGRA~1\Siemens\HIPATH~1\langley\jre\bin\hpi.dll
0x6d680000 - 0x6d68c000 C:\PROGRA~1\Siemens\HIPATH~1\langley\jre\bin\verify.dll
0x6d370000 - 0x6d38d000 C:\PROGRA~1\Siemens\HIPATH~1\langley\jre\bin\java.dll
0x6d6a0000 - 0x6d6af000 C:\PROGRA~1\Siemens\HIPATH~1\langley\jre\bin\zip.dll
0x6d530000 - 0x6d543000 C:\Program Files\Siemens\HiPath Wireless Manager\langley\jre\bin\net.dll
0x71c00000 - 0x71c17000 C:\WINDOWS\system32\WS2_32.dll
0x71bf0000 - 0x71bf8000 C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b61000 C:\WINDOWS\System32\mswsock.dll
0x76ed0000 - 0x76ef9000 C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000 C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000 C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000 C:\WINDOWS\system32\rasadhlp.dll
0x68000000 - 0x6802f000 C:\WINDOWS\system32\rsaenh.dll
0x76920000 - 0x769e4000 C:\WINDOWS\system32\USERENV.dll
0x5f270000 - 0x5f2c9000 C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000 C:\WINDOWS\System32\wshtcpip.dll
0x6d070000 - 0x6d1d7000 C:\Program Files\Siemens\HiPath Wireless Manager\langley\jre\bin\awt.dll
0x73070000 - 0x73097000 C:\WINDOWS\system32\WINSPOOL.DRV
0x76290000 - 0x762ad000 C:\WINDOWS\system32\IMM32.dll
0x77670000 - 0x777a4000 C:\WINDOWS\system32\ole32.dll
0x4d480000 - 0x4d5e5000 C:\Program Files\Siemens\HiPath Wireless Manager\lib\clcapi.dll
0x780c0000 - 0x78121000 C:\WINDOWS\system32\MSVCP60.dll
0x766d0000 - 0x766d9000 C:\WINDOWS\system32\SHFOLDER.dll
0x77d00000 - 0x77d8c000 C:\WINDOWS\system32\OLEAUT32.dll
0x77da0000 - 0x77df2000 C:\WINDOWS\system32\SHLWAPI.dll
0x76cf0000 - 0x76d0a000 C:\WINDOWS\system32\iphlpapi.dll
0x77420000 - 0x77523000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.1830_x-ww_7AE38CCF\comctl32.dll
0x6d660000 - 0x6d666000 C:\Program Files\Siemens\HiPath Wireless Manager\langley\jre\bin\rmi.dll
0x49510000 - 0x49524000 C:\Program Files\Siemens\HiPath Wireless Manager\lib\ws2ping.dll
0x71bb0000 - 0x71bb9000 C:\WINDOWS\system32\WSOCK32.dll
0x76cd0000 - 0x76ce9000 C:\WINDOWS\system32\MPRAPI.dll
0x76df0000 - 0x76e23000 C:\WINDOWS\system32\ACTIVEDS.dll
0x76dc0000 - 0x76de7000 C:\WINDOWS\system32\adsldpc.dll
0x76b80000 - 0x76bae000 C:\WINDOWS\system32\credui.dll
0x7c8d0000 - 0x7d0d3000 C:\WINDOWS\system32\SHELL32.dll
0x76a80000 - 0x76a98000 C:\WINDOWS\system32\ATL.DLL
0x76e30000 - 0x76e3c000 C:\WINDOWS\system32\rtutils.dll
0x5ccf0000 - 0x5ccff000 C:\WINDOWS\system32\SAMLIB.dll
0x770e0000 - 0x771ea000 C:\WINDOWS\system32\SETUPAPI.dll
VM Arguments:
jvm_args: -Xmx1024m -Dcatalina.home=.\\apache\\tomcat -Dmysql.home=.\\mysqldb -Dwebserver.rootdir=.\\apache -Djava.library.path=.\\lib -Dresource_check=9090,8009,8005 -Djava.rmi.server.codebase="http://localhost:9090/classes/ http://localhost:9090/classes/ManagementServer.jar http://localhost:9090/classes/AdventNetSnmp.jar http://localhost:9090/classes/SNMPDebugger.jar http://localhost:9090/classes/AdventNetARUtils.jar http://localhost:9090/classes/AdventNetSAS.jar http://localhost:9090/classes/jta.jar http://localhost:9090/classes/mgmt_service.jar" exit
java_command: <unknown>
Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem
USERNAME=Administrator
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 3, GenuineIntel
S Y S T E M
OS: Windows Server 2003 family Build 3790 Service Pack 1
CPU:total 4 family 15, cmov, cx8, fxsr, mmx, sse, sse2, ht
Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_03-b07) for windows-x86, built on Apr 13 2005 01:49:52 by "java_re" with MS VC++ 6.0
######################################################
Every crash the Java and native frame trace usually report something different, but it is always theFile related operation that causes the crash. Ok, now make it even more bizzar I have another Java standalone application running on this machine that also access files hasno problem, so just wondering if anyone else here had the same problem before. Oh, almost forgot the same program runsfine on asingle processor machine.
Thanks a lot in advance.
