Thanks for the tip.
I don't have that exact process running but found another in my lab which was generating signals
which only had 33 threads so it was a bit easier to monitor.
This was taken on a Solaris 9 sun4u running 1.4.2_12 started with -server and defaulted everything else.
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
22654 adm67 12 0.1 0.0 0.0 18 0.0 2.8 6K 244 18K 329 java/52
22654 adm67 12 0.1 0.0 0.0 18 0.0 2.9 6K 244 18K 333 java/41
22654 adm36 12 0.1 0.0 0.0 48 0.0 3.8 8K 167 21K 789 java/2
22654 adm7.4 19 0.1 0.0 0.0 69 0.0 4.7 15K 60 45K 7K java/1
From multiple pstack blocks, I extracted and pasted below what I figure to be the most interesting traces.
-- lwp# 1 / thread# 1 --
ff2d3aa0 sigvalid (d, 4d58, 0, ff398000, 0, 0) + 70
ff2d3b3c sigaddset (ffbfc514, d, 0, 0, 0, 0) + 4
ff37f7b0 call_user_handler (d, 0, ffbfc5f8, 0, 0, 0) + 1e0
ff37f9b4 sigacthandler (d, 0, ffbfc5f8, ffbfc900, 0, 0) + 64
called from signal handler with signal 13 (SIGPIPE)
ff31fdf4 lwp_cond_wait (37010, 36ff8, 376b0, 0, 0, 0) + 8
fed977ec __1cHMonitorEwait6Mil_i_ (36fc8, 0, 9cba68, 4400, 46a4, 0) + 350
ff100640 __1cHThreadsKdestroy_vm6F_i_ (0, ffbfcae0, ffbfcae4, 0, ff1aa000, 0) + a4
feff374c jni_DestroyJavaVM (ff1ece18, 38120, 38d848, 38168, 37748, 4) + 14c
00013024 main(3810c, 264, 0, 38d848, 0, 26c) + 170c
000118f0 _start(0, 0, 0, 0, 0, 0) + 108
Another trace contained,
-- lwp# 2 / thread# 2 --
ff2c7a28 t_delete (430e68, 2b48, 528, 42ce10, 431458, 0) + dc
ff2c75ec realfree (430f30, 741f4, 0, ff33c000, 0, 0) + c8
ff2c7e5c cleanfree (0, 6, ff3429bc, 8, 840928, 622) + 54
ff2c6f88 _malloc_unlocked (528, 0, 20, ff33c000, 18, 481884) + f4
ff2c6e78 malloc(528, 0, 0, b59130, a, 2c) + 20
fed473a0 __1cCosGmalloc6FI_pv_ (528, f9d13340, f9c00000, 6, 44cd, fed0e718) + 20
ff11a7d0 __1cLvframeArrayIallocate6FpnKJavaThread_ipnNGrowableArray4CpnOcompiledVFramepnLRegisterMap_nFframe_9A9A9A_p0_ (c, 132, b7490, ff1aa000, fe27f958, fe27f948) + a8
fee7d914 __1cODeoptimizationScreate_vframeArray6FpnKJavaThread_nFframe_pnLRegisterMap__pnLvframeArray__ (edfff5c0, 0, f9d13348, f9c35f08, 466520, fed0e718) + 280
fee7fdb0 __1cODeoptimizationKdeoptimize6FpnKJavaThread_nFframe_3pnLRegisterMap__v_ (840c18, fe27fa98, fe27fa88, fe27faf8, edfff560, 0) + ac
fee7f8e8 __1cODeoptimizationQdeoptimize_frame6FpnKJavaThread_pi_v_ (840c18, edfff5c0, fe27fd30, ff1aa000, 44bfc954, fe27fd28) + 224
fee2fd7c __1cMVM_OperationIevaluate6M_v_ (edfff23c, 4800, ff1aa000, 37060, 4dc8cc, fe27fe10) + 8c
fee2f79c __1cIVMThreadSevaluate_operation6MpnMVM_Operation__v_ (ece48, edfff23c, 8cbb0, 32f360, feef4454, 0) + 84
feef44bc __1cIVMThreadEloop6M_v_ (4c00, 4800, 4864, 4400, 47e8, 4000) + 31c
feef4084 __1cIVMThreadDrun6M_v_ (ece48, 2, 40, 0, 40, 0) + 8c
ff0b8614 java_start (ece48, 0, 0, 0, 0, 0) + 134
ff3857b4 _lwp_start (0, 0, 0, 0, 0, 0)
...snip...
-- lwp# 41 / thread# 41 --
ff382840 lmutex_lock (fe3a03d8, fe3a0000, 0, 1a, 4, f9c35f58) + f0
ff37868c find_lwp (29, 1e2f0, 15c2f8, f2f50a38, 1a, f2f509e8) + 3c
ff379d20 thr_getprio (29, ef57f06c, ef57f2e8, 0, ef57f500, 2) + 14
fedf6264 __1cCosTget_native_priority6FpnGThread_pi_nIOSReturn__ (15c2f8, ef57f0d4, f9cf3540, ff1aa000, 392f, 1) + 10
fedf61d4 __1cCosMget_priority6FpnGThread_rnOThreadPriority__nIOSReturn__ (15c2f8, ef57f13c, ef57f2e8, 15c2f8, 3cb9, 1) + 1c
fee1191c __1cGThreadMget_priority6Fp0_nOThreadPriority__ (15c2f8, f9cf2e40, f9c00000, 6, 3cb9, fed3a768) + 8
fee2f934 __1cIVMThreadHexecute6FpnMVM_Operation__v_ (ef57f23c, 15c2f8, ef57f5c0, ff1fdf84, ff1aa000, 0) + a4
fee7f2c8 __1cODeoptimizationNuncommon_trap6FpnKJavaThread_i_pn0ALUnrollBlock__ (15c2f8, fffffffe, fffffffe, 0, f9c099e4, f9c35f58) + 8b0
f9c32e9c ? (fffffffe, f9cf31dc, 0, 1a, 4, f9c35f58)
f9cf3478 ? (f19cc410, f19cc360, 15c2f8, f2f50a38, 1a, f2f509e8)
...snip...
-- lwp# 52 / thread# 52 --
ff31fdf0 lwp_cond_wait (370e0, 370c8, 0)
ff316dfc _lwp_cond_wait (370e0, 370c8, 840c18, ff1aa000, 4dc8cc, edfff138) + c
fed977ec __1cHMonitorEwait6Mil_i_ (37098, 0, b5a0d8, 4400, 46a4, 0) + 350
fee2fafc __1cIVMThreadHexecute6FpnMVM_Operation__v_ (4a44, 840c18, edfff5c0, ff1fdf84, ff1aa000, 0) + 26c
fee7f2c8 __1cODeoptimizationNuncommon_trap6FpnKJavaThread_i_pn0ALUnrollBlock__ (840c18, fffffffe, fffffffe, 0, f9c099e4, f9c35f58) + 8b0
f9c32e9c ? (fffffffe, f9cf31dc, 0, 12, 4, f9c35f58)
f9cf3478 ? (f19d3a08, f19d3968, 840c18, f2f63d48, 12, f2f5faf8)
f9d13490 ? (f2f685f8, f62c0c90, 0, f9c15400, 0, 0)
...snip...
Correlating that to the thread dumps, I notice the Exception is periodically thrown from the Java code,
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.<init>(Throwable.java:195)
at java.lang.Exception.<init>(Exception.java:41)
at java.io.IOException.<init>(IOException.java:40)
at java.net.SocketException.<init>(SocketException.java:29)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
- locked <0xf2f509e8> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.PrintWriter.flush(PrintWriter.java:120)
- locked <0xf2f509e8> (a java.io.OutputStreamWriter)
at com.mycompany.app.Console.println(Console.java:30)
...snip..
So, based on the prstat/pstack info I can conclude this is not identifical
to my previous signals in another program since it wasn't LWP# 1 that
was excessive in it but I have a couple questions...
What is the Signal 13 (SIGPIPE) on the primary thread being used for, why does it have so many more signals then the other LWP's?
What is an Uncommon Trap in JVM terminology and how does that relate to a Java Exception?
Many thanks,
Dan...
