Cause of CodeBuffer memory exhaustion?

We have had a JASIG uPortal application running under Tomcat with JVM 1.5.0_05-b05 for over a year without any JVM crashes. But now, twice within the last 2 weeks, the JVM has crashed because it has run out of CodeBuffer space. There is a bug report on this (6487381), but the bug evaluation section says "VM Crash due to exhaustion of CodeBuffer space, need to provide a softer landing." This tells me that even if the bug is fixed, there will still be an error of some kind.

What is bothering me is that I can see no reason for the CodeBuffer memory exhaustion in the hs_err_pid15701.log file. How can I find out what caused the memory exhaustion? Both of the crashes occurred during periods of light applicaiton load, and the GC stats seem to indicate that there is plenty of memory available.

Any help would be greatly appreciated. The hs_err_pid15701.log output is below. Thanks!

#

# An unexpected error has been detected by HotSpot Virtual Machine:

#

# Internal Error (434F44452255464645520E4350500060 01), pid=15701, tid=15

#

# Java VM: Java HotSpot(TM) Server VM (1.5.0_05-b05 mixed mode)

T H R E A D

Current thread (0x00341318): JavaThread "CompilerThread1" daemon [_thread_in_native, id=15]

Stack: [0x33b02000,0x33b81d70), sp=0x33b7fde0, free space=503k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V [libjvm.so+0x6d2704]

V [libjvm.so+0x42b80c]

V [libjvm.so+0x1b1de0]

V [libjvm.so+0x218688]

V [libjvm.so+0x224330]

V [libjvm.so+0x221abc]

V [libjvm.so+0x217950]

V [libjvm.so+0x27fd24]

V [libjvm.so+0x275c1c]

V [libjvm.so+0x2768d8]

V [libjvm.so+0x332140]

V [libjvm.so+0x2da83c]

V [libjvm.so+0x652d58]

Current CompileTask:

opto:9407 !org.jasig.portal.UserInstance.renderState(Ljavax/servlet/http/HttpServletRequest;Ljav

ax/servlet/http/HttpServletResponse;Lorg/jasig/portal/ChannelManager;Lorg/jasig/portal/i18n/LocaleMa

nager;Lorg/jasig/portal/IUserPreferencesManager;Ljava/lang/Object;)V (2521 bytes)

P R O C E S S

Java Threads: ( => current thread )

0x0135ccd8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266284]

0x014cef80 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266277]

0x02773088 JavaThread "Thread-163071" daemon [_thread_in_native, id=266269]

0x02796250 JavaThread "Thread-163061" daemon [_thread_in_native, id=266254]

0x02d18ae8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266248]

0x03a1b100 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266246]

0x01b8cb30 JavaThread "Thread-163051" daemon [_thread_in_native, id=266235]

0x03a23768 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266227]

0x03a1af40 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266226]

0x00ac6d78 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266190]

0x010bc690 JavaThread "Thread-162977" daemon [_thread_in_native, id=266093]

0x01e1e0f8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266064]

0x01b8ccf0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=266035]

0x00fe6530 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265937]

0x010b9070 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265930]

0x022073a8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265893]

0x02d236f8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265853]

0x0275d630 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265809]

0x02778240 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265716]

0x025ff428 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265583]

0x02337088 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265549]

0x00ac8d80 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265548]

0x01b88dc8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265185]

0x02227808 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265140]

0x0097f438 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265139]

0x02773248 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265057]

0x014c5910 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265024]

0x01e5add8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=265002]

0x0249e028 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264988]

0x012d6f98 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264914]

0x04571230 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264887]

0x015f81f0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264745]

0x0158c258 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264739]

0x00fe5508 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264694]

0x007d4380 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264637]

0x0222ca80 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264623]

0x025c50d0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264606]

0x0249de68 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264596]

0x00f12570 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264565]

0x02606c78 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264548]

0x016ab908 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264481]

0x02d1fa08 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264464]

0x0076c200 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264463]

0x025a6360 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264312]

0x0249fcb0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264293]

0x025a5b98 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264131]

0x0146b9e8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=264110]

0x005c0530 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263911]

0x017713f0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263605]

0x00acb630 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263604]

0x03a02b58 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263564]

0x014c87e8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263563]

0x0264cc18 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263525]

0x025ea728 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263252]

0x03a01250 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263251]

0x022c51c8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263148]

0x03a01090 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263131]

0x00e8d520 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263092]

0x014bff18 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263082]

0x01591e90 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263079]

0x015a8aa0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=263078]

0x02388198 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262738]

0x0235e598 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262712]

0x03a235a8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262540]

0x02e7ac58 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262393]

0x025e70c0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262392]

0x03a00d10 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262358]

0x03a00ed0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262268]

0x010b3658 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262223]

0x025ed920 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262197]

0x01070268 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262196]

0x025e0328 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262011]

0x0222ffd0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=262002]

0x0276c578 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261965]

0x02333d48 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261889]

0x01874f28 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261841]

0x0096a3f0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261777]

0x025eb758 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261755]

0x010b19d0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261651]

0x01e5f058 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261625]

0x0221a1f0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261543]

0x0062e008 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261542]

0x01b8c970 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261534]

0x01580e80 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261435]

0x025e6f00 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261359]

0x00fe92b8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261285]

0x025cd6d8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261264]

0x007912f0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261236]

0x027590c8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=261102]

0x02748138 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=259036]

0x023373c8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=258276]

0x00983b50 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=258096]

0x01e780d0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=258019]

0x02192890 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=257928]

0x002e42a0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=257881]

0x009e23a0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=257856]

0x025f26c0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=257708]

0x02b29128 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=257172]

0x022b5890 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=255735]

0x01baddf8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=255151]

0x022058f8 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=255049]

0x02d47640 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=254806]

0x015d0448 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=253812]

0x025adeb0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=252183]

0x00ed9080 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=250280]

0x00e87c58 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=249544]

0x005ee2a0 JavaThread "JavaMail-EventQueue" daemon [_thread_blocked, id=243050]

0x00510940 JavaThread "TP-Processor160" daemon [_thread_in_native, id=169567]

0x00fcbc48 JavaThread "TP-Processor159" daemon [_thread_blocked, id=169566]

0x01111668 JavaThread "TP-Processor158" daemon [_thread_blocked, id=169565]

0x0212e6e8 JavaThread "TP-Processor157" daemon [_thread_blocked, id=169564]

0x0182ae50 JavaThread "TP-Processor156" daemon [_thread_in_native, id=169558]

0x013c4960 JavaThread "TP-Processor155" daemon [_thread_blocked, id=169557]

0x02359fc8 JavaThread "TP-Processor154" daemon [_thread_blocked, id=169556]

0x010b3498 JavaThread "TP-Processor153" daemon [_thread_in_native, id=169555]

0x00d54918 JavaThread "TP-Processor152" daemon [_thread_blocked, id=169553]

0x01a07020 JavaThread "TP-Processor151" daemon [_thread_in_native, id=169552]

0x012565e8 JavaThread "TP-Processor149" daemon [_thread_blocked, id=169550]

0x00d02d40 JavaThread "TP-Processor148" daemon [_thread_blocked, id=169544]

0x023dab68 JavaThread "TP-Processor146" daemon [_thread_blocked, id=169542]

0x006b0cd8 JavaThread "TP-Processor144" daemon [_thread_in_native, id=169538]

0x012dd0e8 JavaThread "TP-Processor143" daemon [_thread_blocked, id=169537]

0x00774cd8 JavaThread "TP-Processor142" daemon [_thread_blocked, id=169536]

0x02122d78 JavaThread "TP-Processor141" daemon [_thread_blocked, id=169535]

0x01ea2140 JavaThread "TP-Processor133" daemon [_thread_blocked, id=168754]

0x00801200 JavaThread "TP-Processor132" daemon [_thread_in_native, id=168751]

0x017b88b0 JavaThread "TP-Processor126" daemon [_thread_in_native, id=168722]

0x008a8aa8 JavaThread "TP-Processor125" daemon [_thread_in_native, id=168721]

0x011f6bf0 JavaThread "TP-Processor123" daemon [_thread_blocked, id=168703]

0x00d62440 JavaThread "TP-Processor121" daemon [_thread_blocked, id=168701]

0x005423a8 JavaThread "TP-Processor120" daemon [_thread_in_native, id=168511]

0x005f4868 JavaThread "TP-Processor119" daemon [_thread_in_native, id=168510]

0x00fb2308 JavaThread "TP-Processor116" daemon [_thread_blocked, id=168494]

0x00780b38 JavaThread "TP-Processor112" daemon [_thread_blocked, id=168471]

0x018e6d50 JavaThread "TP-Processor111" daemon [_thread_blocked, id=168470]

0x01867210 JavaThread "TP-Processor109" daemon [_thread_blocked, id=168468]

0x024367c0 JavaThread "TP-Processor104" daemon [_thread_blocked, id=167922]

0x019d6b68 JavaThread "TP-Processor103" daemon [_thread_in_native, id=167921]

0x0229f620 JavaThread "TP-Processor100" daemon [_thread_in_native, id=167917]

0x012b2d78 JavaThread "TP-Processor95" daemon [_thread_in_native, id=167912]

0x01071a20 JavaThread "TP-Processor91" daemon [_thread_blocked, id=167908]

0x0299bbe8 JavaThread "TP-Processor86" daemon [_thread_in_native, id=167903]

0x00c77a38 JavaThread "TP-Processor85" daemon [_thread_blocked, id=167902]

0x01910c58 JavaThread "TP-Processor84" daemon [_thread_in_native, id=167901]

0x024b6318 JavaThread "TP-Processor83" daemon [_thread_in_native, id=167900]

0x00e76850 JavaThread "TP-Processor72" daemon [_thread_blocked, id=167889]

0x008771d0 JavaThread "TP-Processor70" daemon [_thread_in_native, id=167887]

0x006b7bf0 JavaThread "TP-Processor68" daemon [_thread_in_native, id=167861]

0x027edd10 JavaThread "TP-Processor59" daemon [_thread_blocked, id=167842]

0x018b0b98 JavaThread "TP-Processor58" daemon [_thread_blocked, id=167841]

0x00ef4690 JavaThread "TP-Processor57" daemon [_thread_blocked, id=167840]

0x017b9078 JavaThread "TP-Processor55" daemon [_thread_blocked, id=167628]

0x00878e58 JavaThread "TP-Processor53" daemon [_thread_blocked, id=167626]

0x007a6950 JavaThread "TP-Processor46" daemon [_thread_in_native, id=160765]

0x010d30a0 JavaThread "TP-Processor41" daemon [_thread_blocked, id=160748]

0x00cdd850 JavaThread "TP-Processor37" daemon [_thread_in_native, id=160723]

0x004061f0 JavaThread "TP-Processor34" daemon [_thread_blocked, id=160687]

0x0116e0e8 JavaThread "Thread-88672" daemon [_thread_blocked, id=143122]

0x0128a780 JavaThread "Thread-88671" daemon [_thread_blocked, id=143121]

0x01b6c758 JavaThread "Thread-88670" daemon [_thread_blocked, id=143120]

0x01749d70 JavaThread "Thread-88669" daemon [_thread_blocked, id=143119]

0x00e23440 JavaThread "Thread-88667" daemon [_thread_blocked, id=143116]

0x019ec860 JavaThread "Thread-88666" daemon [_thread_blocked, id=143115]

0x00e49e20 JavaThread "Thread-88665" daemon [_thread_blocked, id=143114]

0x01239b98 JavaThread "Thread-88663" daemon [_thread_blocked, id=143109]

0x0076d678 JavaThread "Thread-88662" daemon [_thread_blocked, id=143108]

0x00b3b278 JavaThread "Thread-88661" daemon [_thread_blocked, id=143107]

0x00b920c8 JavaThread "Thread-88660" daemon [_thread_blocked, id=143106]

0x00759e18 JavaThread "Thread-88659" daemon [_thread_blocked, id=143105]

0x0234a640 JavaThread "Thread-88658" daemon [_thread_blocked, id=143104]

0x013411f0 JavaThread "Thread-88657" daemon [_thread_blocked, id=143103]

0x00606778 JavaThread "Thread-88656" daemon [_thread_blocked, id=143102]

0x0114f470 JavaThread "Thread-88655" daemon [_thread_blocked, id=143101]

0x0040b588 JavaThread "Thread-88654" daemon [_thread_blocked, id=143100]

0x01aefbf0 JavaThread "Thread-88653" daemon [_thread_blocked, id=143099]

0x02439270 JavaThread "Thread-88652" daemon [_thread_blocked, id=143098]

0x005fec38 JavaThread "Thread-88651" daemon [_thread_blocked, id=143097]

0x01965138 JavaThread "TP-Processor32" daemon [_thread_in_native, id=65561]

0x014196e0 JavaThread "TP-Processor31" daemon [_thread_blocked, id=65560]

0x013d7dc8 JavaThread "TP-Processor29" daemon [_thread_blocked, id=65558]

0x007031e8 JavaThread "Timer-0" daemon [_thread_blocked, id=581]

0x005ecc90 JavaThread "TP-Processor26" daemon [_thread_in_native, id=267]

0x005ecad0 JavaThread "TP-Processor25" daemon [_thread_blocked, id=266]

0x02446190 JavaThread "TP-Processor17" daemon [_thread_blocked, id=121]

0x01130a60 JavaThread "TP-Processor14" daemon [_thread_in_native, id=99]

0x004fd2e8 JavaThread "Thread-47" daemon [_thread_blocked, id=89]

0x00c2a418 JavaThread "Thread-46" daemon [_thread_blocked, id=88]

0x01b489c0 JavaThread "Thread-45" daemon [_thread_blocked, id=87]

0x01acf2e0 JavaThread "TP-Processor10" daemon [_thread_in_native, id=80]

0x00897a18 JavaThread "Thread-39" daemon [_thread_blocked, id=76]

0x009006c0 JavaThread "Thread-38" daemon [_thread_blocked, id=75]

0x008ffc60 JavaThread "Thread-37" daemon [_thread_blocked, id=74]

0x027b8f00 JavaThread "Thread-36" daemon [_thread_blocked, id=73]

0x003f8ce8 JavaThread "Thread-35" daemon [_thread_blocked, id=72]

0x01338c40 JavaThread "Thread-33" daemon [_thread_blocked, id=70]

0x02594ad8 JavaThread "Thread-32" daemon [_thread_blocked, id=69]

0x00d08950 JavaThread "Thread-31" daemon [_thread_blocked, id=68]

0x0185fb98 JavaThread "Thread-30" daemon [_thread_blocked, id=67]

0x00771160 JavaThread "Thread-28" daemon [_thread_blocked, id=52]

0x00cc65b0 JavaThread "Thread-27" daemon [_thread_blocked, id=51]

0x009d50c0 JavaThread "Thread-26" daemon [_thread_blocked, id=50]

0x00ab6318 JavaThread "Thread-25" daemon [_thread_blocked, id=49]

0x009edab8 JavaThread "Thread-24" daemon [_thread_blocked, id=48]

0x00a990d0 JavaThread "Thread-23" daemon [_thread_blocked, id=47]

0x01b4d918 JavaThread "Thread-18" daemon [_thread_blocked, id=40]

0x003a00a0 JavaThread "Thread-16" daemon [_thread_blocked, id=37]

0x017f22c0 JavaThread "Thread-15" daemon [_thread_blocked, id=36]

0x01ac6660 JavaThread "Thread-14" daemon [_thread_blocked, id=35]

0x01ac6440 JavaThread "Thread-13" daemon [_thread_blocked, id=34]

0x003b9c08 JavaThread "Thread-12" daemon [_thread_blocked, id=33]

0x01176860 JavaThread "uPortal ReferenceEntityCache sweeper thread #4" daemon [_thread_blocked, id

=31]

0x01a36580 JavaThread "uPortal ReferenceEntityCache sweeper thread #3" daemon [_thread_blocked, id

=30]

0x010467e8 JavaThread "uPortal ReferenceEntityCache sweeper thread #2" daemon [_thread_blocked, id

=29]

0x01421aa0 JavaThread "Thread-10" daemon [_thread_blocked, id=28]

0x009b3240 JavaThread "uPortal ReferenceEntityCache sweeper thread #1" daemon [_thread_blocked, id

=27]

0x00da46f0 JavaThread "TP-Monitor" daemon [_thread_blocked, id=26]

0x01986f60 JavaThread "TP-Processor4" daemon [_thread_in_native, id=25]

0x011f0808 JavaThread "TP-Processor3" daemon [_thread_blocked, id=24]

0x011f0648 JavaThread "TP-Processor2" daemon [_thread_blocked, id=23]

0x014ec510 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blo

cked, id=21]

0x011ef460 JavaThread "Whirlycache Tuner ( userInfoCache )" daemon [_thread_blocked, id=20]

0x0236d6c8 JavaThread "Whirlycache Tuner ( principalCache )" daemon [_thread_blocked, id=19]

0x02598310 JavaThread "Whirlycache Tuner ( default )" daemon [_thread_blocked, id=18]

0x00342490 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=16]

=>0x00341318 JavaThread "CompilerThread1" daemon [_thread_in_native, id=15]

0x003403e0 JavaThread "CompilerThread0" daemon [_thread_blocked, id=14]

0x0033f4a8 JavaThread "AdapterThread" daemon [_thread_blocked, id=13]

0x003383a0 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12]

0x0032b860 JavaThread "YJPAgent-CPUSampler" daemon [_thread_blocked, id=11]

0x00323220 JavaThread "YJPAgent-RequestListener" daemon [_thread_in_native, id=10]

0x00331d20 JavaThread "YJPAgent-Telemetry" daemon [_thread_blocked, id=9]

0x00312880 JavaThread "Finalizer" daemon [_thread_blocked, id=8]

0x001bf9b0 JavaThread "Reference Handler" daemon [_thread_blocked, id=7]

0x001fe030 JavaThread "main" [_thread_in_native, id=1]

Other Threads:

0x0030af68 VMThread [id=6]

0x00343fe8 WatcherThread [id=17]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap

PSYoungGentotal 310016K, used 250106K [0xc7c00000, 0xdbc00000, 0xf7c00000)

eden space 292352K, 81% used [0xc7c00000,0xd634e4b8,0xd9980000)

from space 17664K, 75% used [0xd9980000,0xda670460,0xdaac0000)

tospace 17344K, 0% used [0xdab10000,0xdab10000,0xdbc00000)

PSOldGentotal 1647808K, used 1572315K [0x37c00000, 0x9c530000, 0xc7c00000)

object space 1647808K, 95% used [0x37c00000,0x97b76df0,0x9c530000)

PSPermGentotal 46848K, used 31224K [0x33c00000, 0x369c0000, 0x37c00000)

object space 46848K, 66% used [0x33c00000,0x35a7e190,0x369c0000)

0x00010000/usr/java/bin/java

0xff360000/usr/lib/libthread.so.1

0xff3a0000/usr/lib/libdl.so.1

0xff280000/usr/lib/libc.so.1

0xff350000/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1

0xfdc00000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/server/libjvm.so

0xff220000/usr/lib/libsocket.so.1

0xff200000/usr/lib/libsched.so.1

0xff1d0000/usr/lib/libCrun.so.1

0xff1a0000/usr/lib/libm.so.1

0xff080000/usr/lib/libnsl.so.1

0xff240000/usr/lib/libw.so.1

0xff160000/usr/lib/libmp.so.2

0xff060000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/native_threads/libhpi.so

0xfe680000/nau/local/opt/yourkit/yjp-5.5.4/bin/solaris-sparc-32/libyjpagent.so

0xfe650000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/libverify.so

0xfe610000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/libjava.so

0xfe5f0000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/libzip.so

0xfe4a0000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/libnet.so

0xfe480000/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/libmanagement.so

0xf7c30000/usr/lib/nss_nis.so.1

0x33be0000/usr/lib/nss_files.so.1

VM Arguments:

jvm_args: -Xms1536m -Xmx3072m -XX:+UseParallelGC -XX:NewSize=320m -XX:NewRatio=3 -verbose:gc -XX:+Pr

intGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -agentpath:/na

u/local/opt/yourkit/yjp-5.5.4/bin/solaris-sparc-32/libyjpagent.so=disablealloc,disablecounts,port=10

111,dir=/nau/local/opt/yourkit -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dj

ava.endorsed.dirs=/nau/local/opt/tomcat/common/endorsed -Dcatalina.base=/nau/local/opt/tomcat -Dcata

lina.home=/nau/local/opt/tomcat -Djava.io.tmpdir=/nau/local/opt/tomcat/temp

java_command: org.apache.catalina.startup.Bootstrap start

Environment Variables:

JAVA_HOME=/usr/java

PATH=/nau/local/opt/uportal/bin:.:/nau/local/bin:/nau/share/bin:/usr/sbin:/usr/bin:/usr/ucb:/opt/bin

:/usr/ccs/bin:/opt/RICHPse/bin:/nau/share/oracle:/nau/share/oracle/product/920/bin:/nau/share/oracle

/product/920/OPatch:/nau/local/opt/ant/bin:/nau/local/opt/ant/bin

LD_LIBRARY_PATH=/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/server:/usr/jdk/instances/jdk1.5.0/jre/lib

/sparc:/usr/jdk/instances/jdk1.5.0/jre/../lib/sparc

SHELL=/bin/ksh

HOSTTYPE=sun4

OSTYPE=solaris

MACHTYPE=sparc

Signal Handlers:

SIGSEGV: [libjvm.so+0x6d3178], sa_mask[0]=0xffbffeff, sa_flags=0x00000004

SIGBUS: [libjvm.so+0x6d3178], sa_mask[0]=0xffbffeff, sa_flags=0x00000004

SIGFPE: [libjvm.so+0x273260], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c

SIGPIPE: [libjvm.so+0x273260], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c

SIGILL: [libjvm.so+0x273260], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c

SIGUSR1: [libjvm.so+0x655038], sa_mask[0]=0x00008000, sa_flags=0x00000008

SIGUSR2: [libjvm.so+0x273260], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c

SIGHUP: [libjvm.so+0x653eb8], sa_mask[0]=0xffbffeff, sa_flags=0x00000004

SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000

SIGQUIT: [libjvm.so+0x653eb8], sa_mask[0]=0xffbffeff, sa_flags=0x00000004

SIGTERM: [libjvm.so+0x653eb8], sa_mask[0]=0xffbffeff, sa_flags=0x00000004

S Y S T E M

OS: Solaris 8 HW 12/02 s28s_hw1wos_06a SPARC

Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.

Assembled 12 December 2002

uname:SunOS 5.8 Generic_108528-23 sun4u (T1 libthread)

rlimit: STACK 8192k, CORE infinity, NOFILE 4096, AS infinity

load average:0.08 0.03 0.03

CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

Memory: 8k page, physical 6291456k(3143232k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_05-b05) for solaris-sparc, built on Aug 26 2005 16:11:55

by unknown with unknown Workshop:0x550

[26596 byte] By [pgazdaa] at [2007-11-26 15:52:31]
# 1

I suspect that you've run out of C malloc space for VM data structures for the CodeBuffer and not Java object heap memory. You could use ps to watch the virtual memory usage of the process.

You are running the 32-bit JVM, and have -Xmx3072m reserving 3GB of the available address space for the heap. You are running 200+ threads, whose stacks also use up address space. You could try shrinking the thread stack size with the -XX:ThreadStackSize= command line option. That might cause StackOverflowError's if you need deep stacks. With tuning, one can run thousands of threads.

At the point of the crash you were using 95% of the Java object heap (the PSOldGen), but if that isn't all live data, you could try shrinking your -Xmx a little to leave room for other VM data structures. You can tell what your live data size is by looking at the GC logs after a full collection. You want the PSOldGen to be at least that large, plus at least enough room for a young generation (being somewhat conservative), plus you need space for the young generation itself. You might have to shrink your young generation a little to make it all fit. (I haven't done the math.) Shrinking your young generation will give you more frequent, but probably shorter, young collection pauses.

PeterKesslera at 2007-7-8 22:12:49 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 2

Peter,

Thanks for your reply. I don't think the heap size should be an issue on this server. We have 6 GB of memory on the server, and only 3 GB allocated for java heap. There are no other significant processes running on the server. Here is a typical memory allocation summary:

Memory: 6144M real, 2796M free, 2683M swap in use, 4211M swap free

So does this mean that there is up to 2796M, not including swap, that is available for C malloc space for VM data structures, or are there some other limits on C malloc space in the JVM that I could adjusted upward to avoid running out of CodeBuffer memory?

pgazdaa at 2007-7-8 22:12:49 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 3
It looks like your using a 32bit VM so your java application will be constrained to 4gigs even though your have 6gigs on the server. Once the total memory used as reported by the os approaches 4gig you will get memory failures as the os will refuse to give the vm memory when it does a
Caffeine0001a at 2007-7-8 22:12:49 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 4
> It looks like your using a 32bit VM so your java> application will be constrained to 4gigs even though> your have 6gigs on the server. Thanks. I wasn't aware of that. Looks like I need to cut back on the heap allocation.
pgazdaa at 2007-7-8 22:12:49 > top of Java-index,Java HotSpot Virtual Machine,Specifications...