How do u read the hs_err_pid<pid>.log file
Hi,
When using hprof or any profiling tool, the hotspot vm crashes and generates the following file. How do we read this file or make any sense of it. Is there any utility, etc..
We are using hotspot jvm 1.4.2_12.
Any help will be appreciated.
BTW here is the latest hs file.
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x6dc487e6, pid=6147, tid=1674062752
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_12-b03 mixed mode)
# Problematic frame:
# C [libhcclsm.so+0x97e6] ossRamboIsFlushing__FP10OSSRamboCB+0x6
#
T H R E A D
Current thread (0x6d0ec870): JavaThread "Thread-10" daemon [_thread_in_native, id=6173]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x686e31e0
Registers:
EAX=0x686e3154, EBX=0xb7fcf4a4, ECX=0x080b4938, EDX=0x08cf42a1
ESP=0x63c81d4c, EBP=0x63c81d4c, ESI=0x63c81f34, EDI=0x63c81f54
EIP=0x6dc487e6, CR2=0x686e31e0, EFLAGS=0x00010212
Top of Stack: (sp=0x63c81d4c)
0x63c81d4c:63c81dac 6dc447eb 686e3154 00000001
0x63c81d5c:00000000 00000000 00000000 00000000
0x63c81d6c:00000000 00000000 00000001 0000007d
0x63c81d7c:00000000 686e3154 e8a53863 00000004
0x63c81d8c:63c81dcc 6dc78801 a86a4000 41d19a0e
0x63c81d9c:08cf42a2 00000021 00000009 00000004
0x63c81dac:63c81ddc 6dc4fa85 080b4930 08cf4298
0x63c81dbc:00000079 63c81f54 63c81ddc 00a7095b
Instructions: (pc=0x6dc487e6)
0x6dc487d6:02 83 e0 01 88 c0 5d c3 89 f6 55 89 e5 8b 45 08
0x6dc487e6:8b 80 8c 00 00 00 c1 e8 03 83 e0 01 88 c0 5d c3
Stack: [0x63c10000,0x63c83000), sp=0x63c81d4c, free space=455k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libhcclsm.so+0x97e6] ossRamboIsFlushing__FP10OSSRamboCB+0x6
C [libhcclsm.so+0x57eb] ossRamboMultiSlotCopy+0x73
C [libhccldt.so+0xa85] ra_writeMessageBlock+0xfd
C [libpiAgent.so+0x16afc] jvmpiAgent_print+0x3c
C [libpiAgent.so+0x19f19] jvmpiAgent_printObjAllocElement+0x2a9
C [libpiAgent.so+0x1f97a]
C [libpiAgent.so+0x20499]
V [libjvm.so+0x35bf4b]
V [libjvm.so+0x44d474]
V [libjvm.so+0x2b8af8]
V [libjvm.so+0x3b760f]
V [libjvm.so+0x2cacdc]
j com.appiancorp.kougar.driver.pooling.a.run()V+8
j java.util.TimerThread.mainLoop()V+221
j java.util.TimerThread.run()V+1
v ~StubRoutines::call_stub
V [libjvm.so+0x2d5a54]
V [libjvm.so+0x3bd559]
V [libjvm.so+0x2d5ca6]
V [libjvm.so+0x2d5576]
V [libjvm.so+0x2d60ef]
V [libjvm.so+0x32fec5]
V [libjvm.so+0x43ab1a]
V [libjvm.so+0x435d33]
V [libjvm.so+0x3bf083]
C [libpthread.so.0+0x53cc]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.appiancorp.kougar.driver.pooling.a.run()V+8
j java.util.TimerThread.mainLoop()V+221
j java.util.TimerThread.run()V+1
v ~StubRoutines::call_stub
P R O C E S S
Java Threads: ( => current thread )
0x081ed380 JavaThread "Thread-40" [_thread_in_native, id=6151]
0x088b1240 JavaThread "http-0.0.0.0-8080-1" daemon [_thread_blocked, id=6196]
0x088b0410 JavaThread "TP-Monitor" daemon [_thread_blocked, id=6195]
0x088af880 JavaThread "TP-Processor10" daemon [_thread_in_native, id=6194]
0x08bf74b8 JavaThread "TP-Processor9" daemon [_thread_blocked, id=6193]
0x095235d0 JavaThread "TP-Processor8" daemon [_thread_blocked, id=6192]
0x09522e40 JavaThread "TP-Processor7" daemon [_thread_blocked, id=6191]
0x09522380 JavaThread "TP-Processor6" daemon [_thread_blocked, id=6190]
0x08c33590 JavaThread "TP-Processor5" daemon [_thread_blocked, id=6189]
0x082d6a70 JavaThread "TP-Processor4" daemon [_thread_blocked, id=6188]
0x09523cf0 JavaThread "TP-Processor3" daemon [_thread_blocked, id=6187]
0x08c34318 JavaThread "TP-Processor2" daemon [_thread_blocked, id=6186]
0x08bb0d18 JavaThread "TP-Processor1" daemon [_thread_blocked, id=6185]
0x6b5ac448 JavaThread "http-0.0.0.0-8080" daemon [_thread_in_native, id=6184]
0x6b515998 JavaThread "JBossLifeThread" [_thread_blocked, id=6183]
0x6b5ac448 JavaThread "http-0.0.0.0-8080" daemon [_thread_in_native, id=6184]
0x6b515998 JavaThread "JBossLifeThread" [_thread_blocked, id=6183]
0x6b2c3f68 JavaThread "Thread-14" daemon [_thread_blocked, id=6181]
0x6b5cdba0 JavaThread "Thread-13" daemon [_thread_blocked, id=6180]
0x6b258ee8 JavaThread "Thread-12" daemon [_thread_blocked, id=6175]
0x6a8829e8 JavaThread "Thread-11" daemon [_thread_blocked, id=6174]
=>0x6d0ec870 JavaThread "Thread-10" daemon [_thread_in_native, id=6173]
0x0826be80 JavaThread "Thread-9" daemon [_thread_blocked, id=6172]
0x08bf9f40 JavaThread "Thread-8" daemon [_thread_blocked, id=6171]
0x08cc18f8 JavaThread "Thread-7" daemon [_thread_blocked, id=6170]
0x08bfa3d8 JavaThread "Thread-6" daemon [_thread_blocked, id=6169]
0x0826b6f0 JavaThread "Thread-5" daemon [_thread_blocked, id=6168]
0x6b024498 JavaThread "Thread-4" daemon [_thread_blocked, id=6167]
0x0889bd88 JavaThread "HSQLDB Timer @ae1cf" daemon [_thread_blocked, id=6166]
0x08767b58 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=6165]
0x08745de0 JavaThread "GC Daemon" daemon [_thread_blocked, id=6164]
0x087362a8 JavaThread "RMI Reaper" [_thread_blocked, id=6163]
0x08741bd0 JavaThread "Thread-2" daemon [_thread_blocked, id=6162]
0x08742940 JavaThread "RMI TCP Accept-4445" daemon [_thread_in_native, id=6161]
0x081296f0 JavaThread "ScannerThread" daemon [_thread_blocked, id=6160]
0x08628238 JavaThread "Thread-0" daemon [_thread_blocked, id=6159]
0x6d013be0 JavaThread "DestroyJavaVM" [_thread_blocked, id=6147]
0x6d002ed0 JavaThread "CompilerThread1" daemon [_thread_blocked, id=6156]
0x6d001db8 JavaThread "CompilerThread0" daemon [_thread_blocked, id=6155]
0x080dbde8 JavaThread "AdapterThread" daemon [_thread_blocked, id=6154]
0x080db178 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6153]
0x080cc928 JavaThread "Finalizer" daemon [_thread_blocked, id=6150]
0x080cb688 JavaThread "Reference Handler" daemon [_thread_blocked, id=6149]
Other Threads:
0x080cabe8 VMThread [id=6148]
0x6d004720 WatcherThread [id=6157]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
def new generationtotal 24448K, used 24448K [0x6e4c0000, 0x6fcc0000, 0x6fcc0000)
eden space 24320K, 100% used [0x6e4c0000, 0x6fc80000, 0x6fc80000)
from space 128K, 100% used [0x6fca0000, 0x6fcc0000, 0x6fcc0000)
tospace 128K,0% used [0x6fc80000, 0x6fc80000, 0x6fca0000)
tenured generationtotal 42796K, used 25127K [0x6fcc0000, 0x7268b000, 0xae4c0000)
the space 42796K, 58% used [0x6fcc0000, 0x71549dd0, 0x71549e00, 0x7268b000)
tenured generationtotal 42796K, used 25127K [0x6fcc0000, 0x7268b000, 0xae4c0000)
the space 42796K, 58% used [0x6fcc0000, 0x71549dd0, 0x71549e00, 0x7268b000)
compacting perm gen total 24832K, used 24633K [0xae4c0000, 0xafd00000, 0xb24c0000)
the space 24832K, 99% used [0xae4c0000, 0xafcce558, 0xafcce600, 0xafd00000)
Dynamic libraries:
00480000-00492000 r-xp 00000000 fd:00 361814/lib/libnsl-2.3.4.so
00492000-00493000 r-xp 00011000 fd:00 361814/lib/libnsl-2.3.4.so
00493000-00494000 rwxp 00012000 fd:00 361814/lib/libnsl-2.3.4.so
00494000-00496000 rwxp 00494000 00:00 0
00a14000-00a2a000 r-xp 00000000 fd:00 360461/lib/ld-2.3.4.so
00a2a000-00a2b000 r-xp 00015000 fd:00 360461/lib/ld-2.3.4.so
00a2b000-00a2c000 rwxp 00016000 fd:00 360461/lib/ld-2.3.4.so
00a2e000-00b54000 r-xp 00000000 fd:00 360463/lib/tls/libc-2.3.4.so
00b54000-00b56000 r-xp 00125000 fd:00 360463/lib/tls/libc-2.3.4.so
00b56000-00b58000 rwxp 00127000 fd:00 360463/lib/tls/libc-2.3.4.so
00b58000-00b5a000 rwxp 00b58000 00:00 0
00b5c000-00b5e000 r-xp 00000000 fd:00 362922/lib/libdl-2.3.4.so
00b5e000-00b5f000 r-xp 00001000 fd:00 362922/lib/libdl-2.3.4.so
00b5f000-00b60000 rwxp 00002000 fd:00 362922/lib/libdl-2.3.4.so
00b62000-00b83000 r-xp 00000000 fd:00 364019/lib/tls/libm-2.3.4.so
00b83000-00b84000 r-xp 00020000 fd:00 364019/lib/tls/libm-2.3.4.so
00b84000-00b85000 rwxp 00021000 fd:00 364019/lib/tls/libm-2.3.4.so
00c68000-00c76000 r-xp 00000000 fd:00 362923/lib/tls/libpthread-2.3.4.so
00c76000-00c77000 r-xp 0000d000 fd:00 362923/lib/tls/libpthread-2.3.4.so
00c77000-00c78000 rwxp 0000e000 fd:00 362923/lib/tls/libpthread-2.3.4.so
00c78000-00c7a000 rwxp 00c78000 00:00 0
00da5000-00dae000 r-xp 00000000 fd:00 364026/lib/libgcc_s-3.4.6-20060404.so.1
00dae000-00daf000 rwxp 00009000 fd:00 364026/lib/libgcc_s-3.4.6-20060404.so.1
08048000-08053000 r-xp 00000000 fd:03 327770/usr/java/j2sdk1.4.2_12/bin/java
08053000-08055000 rwxp 0000a000 fd:03 327770/usr/java/j2sdk1.4.2_12/bin/java
08055000-0b9ed000 rwxp 08055000 00:00 0
637f8000-637f9000 p 637f8000 00:00 0
637f9000-63806000 rwxp 637f9000 00:00 0
63806000-63809000 p 63806000 00:00 0
63809000-63879000 rwxp 63809000 00:00 0
63879000-6387a000 p 63879000 00:00 0
6387a000-63887000 rwxp 6387a000 00:00 0
63887000-6388a000 p 63887000 00:00 0
6388a000-638fa000 rwxp 6388a000 00:00 0
638fa000-638fb000 p 638fa000 00:00 0
638fb000-63908000 rwxp 638fb000 00:00 0
63908000-6390b000 rwxp 63908000 00:00 0
6390b000-6397b000 rwxp 6390b000 00:00 0
... same data as above..
VM Arguments:
jvm_args: -Duser.timezone=GMT -Duser.language= -Xmx1024m -Dsun.rmi.dgc.client.gcInterval=86400000 -Dsun.rmi.dgc.server.gcInterval=86400000 -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:gc.txt -XX:NewSize=24m -XX:MaxNewSize=24m -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=150 -Djava.awt.headless=true -Dprogram.name=run.sh -XrunpiAgent:server=enabled -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/home/appian/jboss-4.0.2/lib/endorsed
java_command: org.jboss.Main
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java/j2sdk1.4.2_12
PATH=/usr/java/j2sdk1.4.2_12/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/appian/bin:/usr/local/apache-ant-1.6.5/bin
LD_LIBRARY_PATH=/usr/java/j2sdk1.4.2_12/jre/lib/i386/server:/usr/java/j2sdk1.4.2_12/jre/lib/i386:/usr/java/j2sdk1.4.2_12/jre/../lib/i386
SHELL=/bin/bash
S Y S T E M
OS:Red Hat Enterprise Linux ES release 4 (Nahant Update 5)
uname:Linux 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 81920, NOFILE 1024, AS infinity
load average:325215078481544067418021531507253339114023570572081208587916205359104.00 131198811606042569760500027299787024097352946895170280943343508464786882673889726356210730355696686997119451320235216116321080642087335108750767273038475800621827641579806615210529115622778654545791307446324043502938668805095862304768.00 0.00
CPU:total 4 family 15, cmov, cx8, fxsr, mmx, sse, sse2
Memory: 4k page, physical 1012k(60k free), swap 511k(511k free)
vm_info: Java HotSpot(TM) Server VM (1.4.2_12-b03) for linux-x86, built on May 9 2006 12:16:45 by unknown with unknown compiler
