SUN JES 8.1 Q2UR2: Intermittently application server jvm crashed.
Application server jvm intermittently crashed but due to the feature in SUN JES 8.1 ur2 the application server will auto restarts the service. Most of the JVM crashes were due to memory corruption.
The following are SUN findings:
1)Multithreading issue that causes 2 threads writing on the same memory block
2)Based on dtrace, it indicated that there was a race condition happens but the jvm doesn抰 crash during the occurrence.
Questions:
1) Is it possible that I can use dtrace to monitor the memory block or to find out the root cause?
System information:
Oracle:
DriverVersion: [10.2.0.1.0]
DriverMajorVersion: [10]
DriverMinorVersion: [2]
DriverName: [Oracle JDBC driver]
URL: [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL = TCP) (HOST = XXXXXX) (PORT = 1524)) (ADDRESS=(PROTOCOL = TCP) (HOST = XXXXXX) (PORT = 1524))) (CONNECT_DATA = (SID = CP_P)) (source_route=yes))]
UserName: [CP_P1]
Database Product Name: [Oracle]
Database Product Version: [Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production]
Database Product Major: [9]
Database Product Minor: [2]
Drivers: Type 4
JDK
JDBC Major: [10]
JDBC Minor: [2]
The JDK version is 1.5.0_02
SUN JES Application Server
Sun Java System Application Server Enterprise Edition 8.1_02 running under solaris 10
[1548 byte] By [
csgan97a] at [2007-10-3 11:08:00]
> Most of the
> JVM crashes were due to memory corruption.
>
And how do you know that?
> The following are SUN findings:
Meaning Sun (the company) and not you or someone else determined this?
> 1) Multithreading issue that causes 2 threads writing
> on the same memory block
So? That might indicate a bug in your code. By itself it doesn't mean that it caused the crash.
> 2) Based on dtrace, it indicated that there was a
> race condition happens but the jvm doesn抰 crash
> during the occurrence.
Again so? I wouldn't expect that to cause a crash.
> DriverName: [Oracle JDBC driver]
> URL: [jdbc:oracle:thin
If it is memory corruption then it would be a JVM bug not this driver because it is java only.
If you have any JNI or 3rd party libraries that use JNI then those are the most likely source.
> Most of the
> JVM crashes were due to memory corruption.
>
And how do you know that?
csgan97: Based on sun support input where its due to either trying to read or write to a memory (pblock) that causes the crash.
could identify as a pblock issue
-- lwp# 102 / thread# 102 -- fef40f90 _lwp_kill (6, 0, fef24a98, ffffffff, fef68284, 6) + 8
feebff98 abort(4a600, 1, febbeaac, a83f0, fef6b298, 0) + 110
febb9a9c __1cCosFabort6Fi_v_ (1, 4e10, 10c5b4, 1cb, fecc6000, 4c00) + 58 fec325a0 ? (0, fece3ee0, 4000, febbd19c, fec37eee, febbd19c) + 78b5c
fe9ff2d8 JVM_handle_solaris_signal (b, 5931fa70, 5931f7b8, 5000, 4,
184c128) +
9e8
fef3fec8 __sighndlr (b, 5931fa70, 5931f7b8, fe9fe8d0, 0, 1) + c
fef34ff4 call_user_handler (b, ffbffeff, c, 0, 5bb37c00, 5931f7b8) + 3b8 5f2b08fc pblock_kpinsert (4efa78, 1246c28, 1246520, 2c00, 7398ae8,
2e64) + 17c
5f2b0b28 pblock_kvinsert (4efa78, 1246900, 12, 1246520, ff00, 80808080)
+ 108
5f2d5938 INTservact_objset_uri2path (1237c80, 1237cf8, e436b0, 30f4, 680ac6a8,
5f3e5f54) + e4
5f2d77a8 INTservact_handle_processed (1237c80, 1237cf8, 1237c80, 1237cf8, 680ea948, 10ddda0) + 4c 5f31a0ec __1cLHttpRequestUUnacceleratedRespond6Mpc_v_ (1237be0, 5f3ed614, 3114, 50, 1237cf8, 1237c80) + 3c8 5f3197dc __1cLHttpRequestNHandleRequest6MpnGnetbuf__i_ (1237be0, 12442d8, 1246430, 1246410, 1000, 1244338) + 62c
5f317bc4 __1cNDaemonSessionDrun6M_v_ (1234430, 1000, 5f3f3254, 0, 0,
5f3f320c)
+ 17c
627e6dbc ThreadMain (1234430, 1240848, 3, 0, 1, 0) + 24
680d0028 _pt_root (1240848, 59320000, 0, 0, 40000, 680e8d70) + d0 fef3fd9c _lwp_start (0, 0, 0, 0, 0, 0)
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x5f2b08fc, pid=16489, tid=102 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_02-b09 mixed mode) # Problematic frame:
# C [libns-httpd40.so+0x308fc] pblock_kpinsert+0x17c #
> The following are SUN findings:
Meaning Sun (the company) and not you or someone else determined this?
csgan97: From Sun support engineer
> 1) Multithreading issue that causes 2 threads writing
> on the same memory block
So? That might indicate a bug in your code. By itself it doesn't mean that it caused the crash.
csgan97: If there is a bug, we would like to identify it. All findings are hypothesis and we still dont have evident to really prove that is multithreading issue. Is it possible to use dtrace to track?
> 2) Based on dtrace, it indicated that there was a
> race condition happens but the jvm doesn抰 crash
> during the occurrence.
Again so? I wouldn't expect that to cause a crash.
csgan97: Why? During the jvm crashed, the dtrace monitoring doesnt indicate any race condition. It only indicated twice but during that time the jvm doesnt crash.
> DriverName: [Oracle JDBC driver]
> URL: [jdbc:oracle:thin
If it is memory corruption then it would be a JVM bug not this driver because it is java only.
If you have any JNI or 3rd party libraries that use JNI then those are the most likely source.
csgan97: A far as we know there isnt any JNI or 3rd party libraries that use JNI.
