Sudden JVM chrashes (SIGSEGV) running JBoss 4.0.5 GA and JDK 1.5.0_11-b03
After having upgraded one of our servers to JBoss 4.0.5 GA and JDK 1.5.0_11-b03, we are experiencing
a lot of JVM chrashes. With just a few hours interval we are getting SIGSEGVs and they all indicates
a problematic frame somewhere inside libjvm.so:
libjvm.so+0x1caa74 (9 incidents)
libjvm.so+0x1c7b34 (3 incidents)
libjvm.so+0x6af4c4(1 incident)
I am of course eager to know the cause of these incidents. Below is one of the error logs. I cannot find
any exceptions in our ordinary logs that are connected to these incidents. Could the SIGSEGVs be
caused by other things than a bug in the JVM?
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xfe9caa74, pid=22832, tid=11560
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x1caa74]
#
T H R E A D
Current thread (0x0198f228): JavaThread"Thread-11075" daemon [_thread_in_vm, id=11560]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000
Registers:
O0=0xf6644755 O1=0xf6644755 O2=0x2547f058 O3=0x00000000
O4=0x00000000 O5=0x00000001 O6=0x2547ede8 O7=0xfe9ca8ac
G1=0xfe9ca8ac G2=0xf6644778 G3=0x099cb8aa G4=0x00009400
G5=0x2547f0b8 G6=0x00000000 G7=0x2b597000 Y=0x00000000
PC=0xfe9caa74 nPC=0xfe9caa78
Top of Stack: (sp=0x2547ede8)
0x2547ede8:fefd8000 00000000 0175e3a0 0198f2d0
0x2547edf8:00000000 f6644710 2547f054 2547f0b8
0x2547ee08:0175e394 00000000 0198f228 00007c00
0x2547ee18:0175e3a0 00c9f7b0 2547f0b8 f8c09ff8
0x2547ee28:ff0273e8 00000002 00000001 0198f228
0x2547ee38:00000000 00000000 2547efb0 2547ef20
0x2547ee48:fefd8000 2547e700 faff01dc 00000000
0x2547ee58:fea85b40 fafdc2b4 f8c38840 000249f0
Instructions: (pc=0xfe9caa74)
0xfe9caa64:81 e8 00 00 c4 06 20 00 37 00 00 1f ac 07 bf 9c
0xfe9caa74:da 06 60 00 86 06 e1 98 c8 00 a0 08 e4 04 00 03
Stack: [0x25400000,0x25480000), sp=0x2547ede8, free space=507k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1caa74]
j tbs.messaging.util.socket.io.WrappedInputStream.close()V+21
j java.io.FilterInputStream.close()V+4
j org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.close()V+11
j org.apache.xerces.impl.io.UTF8Reader.close()V+4
j org.apache.xerces.impl.XMLEntityManager.closeReaders()V+23
j org.apache.xerces.parsers.XML11Configuration.cleanup()V+4
j org.apache.xerces.parsers.XML11Configuration.parse(Lorg/apache/xerces/xni/parser/XMLInputSource;)V+78
j org.apache.xerces.parsers.XMLParser.parse(Lorg/apache/xerces/xni/parser/XMLInputSource;)V+9
j org.apache.xerces.parsers.DOMParser.parse(Lorg/xml/sax/InputSource;)V+43
v ~C2IAdapter
J tbs.messaging.util.socket.XMLClient.queryServer(Ljava/lang/String;)Lorg/w3c/dom/Document;
v ~I2CAdapter
j tbs.messaging.util.socket.XMLClient.queryServer(Lorg/w3c/dom/Document;)Lorg/w3c/dom/Document;+11
j tbs.messaging.mobileoffice.MailService.logout()Lorg/w3c/dom/Document;+82
j tbs.messaging.mobileoffice.MailService.cleanup()V+18
j tbs.messaging.mobileoffice.MOUser$MailserviceCleanup.run()V+16
v ~StubRoutines::call_stub
V [libjvm.so+0x19b95c]
V [libjvm.so+0x2c1a24]
V [libjvm.so+0x2e1010]
V [libjvm.so+0x2dcbac]
V [libjvm.so+0x67a7e0]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j tbs.messaging.util.socket.io.WrappedInputStream.close()V+21
j java.io.FilterInputStream.close()V+4
j org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.close()V+11
j org.apache.xerces.impl.io.UTF8Reader.close()V+4
j org.apache.xerces.impl.XMLEntityManager.closeReaders()V+23
j org.apache.xerces.parsers.XML11Configuration.cleanup()V+4
j org.apache.xerces.parsers.XML11Configuration.parse(Lorg/apache/xerces/xni/parser/XMLInputSource;)V+78
j org.apache.xerces.parsers.XMLParser.parse(Lorg/apache/xerces/xni/parser/XMLInputSource;)V+9
j org.apache.xerces.parsers.DOMParser.parse(Lorg/xml/sax/InputSource;)V+43
v ~C2IAdapter
J tbs.messaging.util.socket.XMLClient.queryServer(Ljava/lang/String;)Lorg/w3c/dom/Document;
v ~I2CAdapter
j tbs.messaging.util.socket.XMLClient.queryServer(Lorg/w3c/dom/Document;)Lorg/w3c/dom/Document;+11
j tbs.messaging.mobileoffice.MailService.logout()Lorg/w3c/dom/Document;+82
j tbs.messaging.mobileoffice.MailService.cleanup()V+18
j tbs.messaging.mobileoffice.MOUser$MailserviceCleanup.run()V+16
v ~StubRoutines::call_stub
P R O C E S S
Java Threads: ( => current thread )
0x00e75588 JavaThread"Thread-11630" daemon [_thread_blocked, id=12148]
0x00c89bb8 JavaThread"Thread-11625" daemon [_thread_blocked, id=12143]
0x02119a68 JavaThread"Thread-11611" daemon [_thread_blocked, id=12129]
0x02b408c0 JavaThread"RMI RenewClean-[10.122.115.82:4444]" daemon [_thread_blocked, id=12097]
0x0285b128 JavaThread"RMI ConnectionExpiration-[10.122.115.82:1098]" daemon [_thread_blocked, id=12096]
0x0113ac60 JavaThread"RMI TCP Connection(176)-10.122.115.82" daemon [_thread_in_native, id=12095]
0x0273ba28 JavaThread"RMI RenewClean-[10.122.115.82:1098]" daemon [_thread_blocked, id=12094]
0x01089670 JavaThread"JBoss System Threads(1)-49" daemon [_thread_blocked, id=12093]
0x00729778 JavaThread"RMI TCP Connection(175)-10.122.115.82" daemon [_thread_in_native, id=12089]
0x0247a2c8 JavaThread"Thread-11578" daemon [_thread_blocked, id=12086]
0x016f20c8 JavaThread"RMI ConnectionExpiration-[10.122.115.82:1098]" daemon [_thread_blocked, id=11872]
=>0x0198f228 JavaThread"Thread-11075" daemon [_thread_in_vm, id=11560]
0x0108ea80 JavaThread"Thread-9230" daemon [_thread_in_native, id=9635]
0x00a742e8 JavaThread"Thread-9228" daemon [_thread_in_native, id=9633]
0x03c1ff90 JavaThread"Thread-6281" daemon [_thread_in_native, id=6559]
0x003b8880 JavaThread"Thread-420" daemon [_thread_in_native, id=475]
0x0107c740 JavaThread"Thread-213" daemon [_thread_in_native, id=255]
0x01691f30 JavaThread"RMI LeaseChecker" daemon [_thread_blocked, id=248]
0x013ddc68 JavaThread"TP-Processor56" daemon [_thread_blocked, id=216]
0x025d0fa8 JavaThread"TP-Processor55" daemon [_thread_blocked, id=215]
0x01c7f4f8 JavaThread"TP-Processor54" daemon [_thread_blocked, id=214]
0x00bb9250 JavaThread"TP-Processor53" daemon [_thread_blocked, id=213]
0x00bb7ea0 JavaThread"TP-Processor52" daemon [_thread_blocked, id=205]
0x00bb7228 JavaThread"TP-Processor51" daemon [_thread_blocked, id=204]
0x010906a0 JavaThread"TP-Processor50" daemon [_thread_blocked, id=203]
0x01c47118 JavaThread"TP-Processor49" daemon [_thread_blocked, id=202]
0x01d02ab0 JavaThread"TP-Processor48" daemon [_thread_blocked, id=188]
0x01d01f70 JavaThread"TP-Processor47" daemon [_thread_blocked, id=187]
0x015df8e8 JavaThread"TP-Processor46" daemon [_thread_blocked, id=186]
0x025fc258 JavaThread"TP-Processor45" daemon [_thread_blocked, id=185]
0x010df920 JavaThread"TP-Processor44" daemon [_thread_blocked, id=177]
0x01c7f798 JavaThread"TP-Processor43" daemon [_thread_blocked, id=176]
0x01030e80 JavaThread"TP-Processor42" daemon [_thread_blocked, id=175]
0x003bc550 JavaThread"TP-Processor41" daemon [_thread_blocked, id=174]
0x017fb9d0 JavaThread"Thread-139" daemon [_thread_in_native, id=173]
0x0124e228 JavaThread"Thread-138" daemon [_thread_in_native, id=172]
0x010deda0 JavaThread"Thread-133" daemon [_thread_in_native, id=167]
0x003b9508 JavaThread"TP-Processor40" daemon [_thread_blocked, id=165]
0x0158a418 JavaThread"TP-Processor39" daemon [_thread_blocked, id=164]
0x02bc18f8 JavaThread"TP-Processor38" daemon [_thread_in_native, id=163]
0x01c664a8 JavaThread"TP-Processor37" daemon [_thread_blocked, id=162]
0x02323370 JavaThread"Thread-125" daemon [_thread_in_native, id=159]
0x0173ae28 JavaThread"Thread-123" daemon [_thread_in_native, id=157]
0x025f8f68 JavaThread"TP-Processor36" daemon [_thread_blocked, id=150]
0x01b05f50 JavaThread"TP-Processor35" daemon [_thread_in_native, id=149]
0x01a937c0 JavaThread"TP-Processor34" daemon [_thread_in_native, id=148]
0x01ea97e8 JavaThread"TP-Processor33" daemon [_thread_blocked, id=147]
0x022f5e58 JavaThread"TP-Processor32" daemon [_thread_in_native, id=140]
0x022f50f8 JavaThread"TP-Processor31" daemon [_thread_blocked, id=139]
0x02511730 JavaThread"TP-Processor30" daemon [_thread_blocked, id=138]
0x025109d0 JavaThread"TP-Processor29" daemon [_thread_blocked, id=137]
0x019ad330 JavaThread"TP-Processor28" daemon [_thread_blocked, id=125]
0x01729230 JavaThread"TP-Processor27" daemon [_thread_blocked, id=124]
0x0212c438 JavaThread"TP-Processor26" daemon [_thread_blocked, id=123]
0x02134430 JavaThread"TP-Processor25" daemon [_thread_blocked, id=122]
0x009d1df8 JavaThread"TP-Processor24" daemon [_thread_blocked, id=104]
0x01ef1fe0 JavaThread"TP-Processor23" daemon [_thread_blocked, id=103]
0x01ef1430 JavaThread"TP-Processor22" daemon [_thread_blocked, id=102]
0x007288b0 JavaThread"TP-Processor21" daemon [_thread_in_native, id=101]
0x01362f20 JavaThread"TP-Processor20" daemon [_thread_in_native, id=94]
0x01362be0 JavaThread"TP-Processor19" daemon [_thread_blocked, id=93]
0x01362900 JavaThread"TP-Processor18" daemon [_thread_in_native, id=92]
0x01362738 JavaThread"TP-Processor17" daemon [_thread_in_native, id=91]
0x0103d3e0 JavaThread"TP-Processor16" daemon [_thread_blocked, id=77]
0x02574a08 JavaThread"TP-Processor15" daemon [_thread_blocked, id=76]
0x02573fe0 JavaThread"TP-Processor14" daemon [_thread_in_native, id=75]
0x026dfea8 JavaThread"TP-Processor13" daemon [_thread_blocked, id=74]
0x02323958 JavaThread"Thread-39" daemon [_thread_in_native, id=73]
0x02160b20 JavaThread"Thread-33" daemon [_thread_in_native, id=67]
0x020b0c00 JavaThread"Thread-31" daemon [_thread_in_native, id=65]
0x023fd9e0 JavaThread"TP-Processor12" daemon [_thread_in_native, id=62]
0x01cb36b0 JavaThread"TP-Processor11" daemon [_thread_in_native, id=61]
0x011f43a8 JavaThread"TP-Processor10" daemon [_thread_in_native, id=60]
0x02624468 JavaThread"TP-Processor9" daemon [_thread_in_native, id=59]
0x0261a1d8 JavaThread"Thread-24" daemon [_thread_in_native, id=58]
0x005711f0 JavaThread"Thread-22" daemon [_thread_in_native, id=56]
0x0178dbf0 JavaThread"TP-Processor8" daemon [_thread_in_native, id=53]
0x00d45618 JavaThread"TP-Processor7" daemon [_thread_in_native, id=52]
0x01c80a50 JavaThread"TP-Processor6" daemon [_thread_in_native, id=51]
0x01cedd58 JavaThread"TP-Processor5" daemon [_thread_in_native, id=50]
0x01ad15f8 JavaThread"http-0.0.0.0-8080-1" daemon [_thread_blocked, id=46]
0x0202e9c8 JavaThread"TP-Monitor" daemon [_thread_blocked, id=45]
0x028bb968 JavaThread"TP-Processor4" daemon [_thread_in_native, id=44]
0x0142dba8 JavaThread"TP-Processor3" daemon [_thread_in_native, id=43]
0x01091bc0 JavaThread"TP-Processor2" daemon [_thread_in_native, id=42]
0x01ad0780 JavaThread"TP-Processor1" daemon [_thread_in_native, id=41]
0x0142e2c8 JavaThread"http-0.0.0.0-8080" daemon [_thread_in_native, id=40]
0x0142d5d8 JavaThread"JBossLifeThread" [_thread_blocked, id=39]
0x00a2e388 JavaThread"Thread-6" daemon [_thread_in_native, id=38]
0x010a9e88 JavaThread"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:1/7f2340c31bc8650d2ff2929db3464312 destination=QUEUE.B messageSelector=null Local Create] id=1" [_thread_blocked, id=37]
0x02a2f850 JavaThread"UILServerILService Accept Thread" [_thread_in_native, id=36]
0x01f6cc50 JavaThread"TimeoutFactory-1" daemon [_thread_blocked, id=35]
0x00bbdde8 JavaThread"JCA PoolFiller" [_thread_blocked, id=34]
0x00e9ef40 JavaThread"Timer-2" daemon [_thread_blocked, id=33]
0x02161630 JavaThread"HSQLDB Timer @1a9b4a4" daemon [_thread_blocked, id=32]
0x0121e4a8 JavaThread"WorkManager(3)-1" daemon [_thread_blocked, id=31]
0x0140e358 JavaThread"Thread-3" daemon [_thread_blocked, id=30]
0x00490d78 JavaThread"JBossMQ Cache Reference Softner" daemon [_thread_blocked, id=29]
0x00659af0 JavaThread"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=28]
0x0104ac90 JavaThread"SubscriptionWatchDog" [_thread_blocked, id=27]
0x011685e8 JavaThread"PooledInvokerAcceptor#0-4445" [_thread_in_native, id=26]
0x011171a8 JavaThread"RMI TCP Accept-4444" daemon [_thread_in_native, id=25]
0x014c49e0 JavaThread"TimeoutFactory-0" daemon [_thread_blocked, id=24]
0x01143f70 JavaThread"JBoss System Threads(1)-2" daemon [_thread_in_native, id=23]
0x01791ed0 JavaThread"GC Daemon" daemon [_thread_blocked, id=22]
0x0066c0d0 JavaThread"RMI Reaper" [_thread_blocked, id=21]
0x015b22f8 JavaThread"Timer-1" daemon [_thread_blocked, id=20]
0x00fc9640 JavaThread"RMI TCP Accept-1098" daemon [_thread_in_native, id=19]
0x0070c880 JavaThread"JBoss System Threads(1)-1" daemon [_thread_in_native, id=18]
0x00718420 JavaThread"ScannerThread" daemon [_thread_blocked, id=17]
0x00336a10 JavaThread"Timer-0" daemon [_thread_blocked, id=16]
0x000378c8 JavaThread"DestroyJavaVM" [_thread_blocked, id=1]
0x0018a360 JavaThread"Low Memory Detector" daemon [_thread_blocked, id=13]
0x00189280 JavaThread"CompilerThread1" daemon [_thread_blocked, id=12]
0x00187e00 JavaThread"CompilerThread0" daemon [_thread_blocked, id=11]
0x00186f70 JavaThread"AdapterThread" daemon [_thread_blocked, id=10]
0x00186090 JavaThread"Signal Dispatcher" daemon [_thread_blocked, id=9]
0x001854a0 JavaThread"Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=8]
0x00179a10 JavaThread"Finalizer" daemon [_thread_blocked, id=7]
0x001782c0 JavaThread"Reference Handler" daemon [_thread_blocked, id=6]
Other Threads:
0x001761e0 VMThread [id=5]
0x0018b5c0 WatcherThread [id=14]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
parnew generationtotal 8128K, used 3955K [0x34800000, 0x35000000, 0x35000000)
eden space 8064K, 49% used [0x34800000, 0x34bdccf8, 0x34fe0000)
from space 64K,0% used [0x34ff0000, 0x34ff0000, 0x35000000)
tospace 64K,0% used [0x34fe0000, 0x34fe0000, 0x34ff0000)
concurrent mark-sweep generation total 1040384K, used 141102K [0x35000000, 0x74800000, 0xf4800000)
concurrent-mark-sweep perm gen total 55504K, used 33152K [0xf4800000, 0xf7e34000, 0xf8800000)
Dynamic libraries:
0x00010000/local/java/sdk/1.5.0_11-b03/bin/java
0xff380000/usr/lib/libthread.so.1
0xff3fa000/usr/lib/libdl.so.1
0xff280000/usr/lib/libc.so.1
0xff3b0000/usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1
0xfe800000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/server/libjvm.so
0xff250000/usr/lib/libsocket.so.1
0xff230000/usr/lib/libsched.so.1
0xff200000/usr/lib/libCrun.so.1
0xff1b0000/usr/lib/libm.so.1
0xff080000/usr/lib/libnsl.so.1
0xff180000/usr/lib/libmp.so.2
0xff060000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/native_threads/libhpi.so
0xfe7e0000/usr/lib/nss_files.so.1
0xfe7a0000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/libverify.so
0xfe750000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/libjava.so
0xfe730000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/libzip.so
0xfe310000/usr/lib/locale/iso_8859_1/iso_8859_1.so.2
0xfafd0000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/libnet.so
0xfaee0000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/libmanagement.so
0x2b3e0000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/libnio.so
0x2b3c0000/usr/lib/librt.so.1
0x2b3a0000/usr/lib/libaio.so.1
0x2b2e0000/usr/lib/libmd5.so.1
0x2b2c0000/usr/lib/libsendfile.so.1
0x2b0d0000/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/librmi.so
VM Arguments:
jvm_args: -Dprogram.name=run.sh -Dslb.host.id=2 -Xmx3072m -Xms1024m -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseParNewGC -XX:NewRatio=1 -Djava.endorsed.dirs=/local/jboss/current/lib/endorsed
java_command: org.jboss.Main --configuration=moffice
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/local/java/current
PATH=/local/java/current/bin:/local/gnu/qt/3.1.2/bin:/local/perl/5.8.3/bin:/local/gnu/bin:/local/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/sbin:/local/X11R6/bin:/local/X11/bin:/usr/ucb
LD_LIBRARY_PATH=/local/java/sdk/1.5.0_11-b03/jre/lib/sparc/server:/local/java/sdk/1.5.0_11-b03/jre/lib/sparc:/local/java/sdk/1.5.0_11-b03/jre/../lib/sparc
SHELL=/local/gnu/bin/bash
HOSTTYPE=sparc
OSTYPE=solaris2.6
MACHTYPE=sparc-sun-solaris2.6
Signal Handlers:
SIGSEGV: [libjvm.so+0x709bb4], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x709bb4], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x274474], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x274474], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x274474], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0x67cf24], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0x274474], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0x67bb74], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x67bb74], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x67bb74], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x67bb74], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
S Y S T E M
OS:Solaris 9 8/03 s9s_u4wos_08a SPARC
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 13 June 2003
uname:SunOS 5.9 Generic_122300-02 sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:1.50 1.52 1.79
CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 6291456k(2953816k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_11-b03)for solaris-sparc, built on Dec 15 2006 01:18:11 by unknown with unknown Workshop:0x550
