Full GC unloads sun.reflect Full GC takes long

I have an app that runs fine for several minutes and between two GC cycles, the young generation heap cannot be cleaned. When this happens a Full GC is invoked causing the JVM to unload sun.reflect classes that appear to be obfuscated and only used once. I am thinking the sun reflect classes are filling the Tenured heap causing this to happen. The hardware is in a secure zone so I am looking for a way to determine what is filling up the heap. Is there a way to determine how many times these reflect classes are used and how to de-reference and unload them without causing the long pause? How can I obtain dump of the heap? Concurrent GC and Parallel GC seems to consume more CPU than the app can afford. The service time for the servlet is in the 150 milli-seconds and seems to generate a fair, but reasonable, amount of garbage per request.

3861.135: [GC 3861.136: [DefNew

Desired survivor size 5505024 bytes, new threshold 16 (max 31)

- age1:3675192 bytes,3675192 total

- age2:489392 bytes,4164584 total

- age3:219352 bytes,4383936 total

- age4:72736 bytes,4456672 total

- age5:24264 bytes,4480936 total

- age6:7512 bytes,4488448 total

- age7:102768 bytes,4591216 total

- age8:131752 bytes,4722968 total

- age9:33544 bytes,4756512 total

- age 10:68504 bytes,4825016 total

- age 11:124048 bytes,4949064 total

- age 12:209808 bytes,5158872 total

- age 13:31840 bytes,5190712 total

- age 14:177448 bytes,5368160 total

- age 15:24360 bytes,5392520 total

- age 16:226672 bytes,5619192 total

- age 17:41592 bytes,5660784 total

:178356K->5528K(183808K), 0.1725869 secs] 427582K->254754K(436992K), 0.1728928 secs]

3864.924: [GC 3864.924: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:3396600 bytes,3396600 total

- age2:266688 bytes,3663288 total

- age3:451608 bytes,4114896 total

- age4:106472 bytes,4221368 total

- age5:72736 bytes,4294104 total

- age6:24264 bytes,4318368 total

- age7:7512 bytes,4325880 total

- age8:102256 bytes,4428136 total

- age9:131752 bytes,4559888 total

- age 10:33544 bytes,4593432 total

- age 11:68504 bytes,4661936 total

- age 12:124048 bytes,4785984 total

- age 13:209808 bytes,4995792 total

- age 14:31840 bytes,5027632 total

- age 15:177448 bytes,5205080 total

- age 16:24360 bytes,5229440 total

:178584K->5106K(183808K), 0.1391098 secs] 427810K->254594K(436992K), 0.1393726 secs]

3868.648: [GC 3868.648: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:3508552 bytes,3508552 total

- age2:174296 bytes,3682848 total

- age3:146728 bytes,3829576 total

- age4:407376 bytes,4236952 total

- age5:106472 bytes,4343424 total

- age6:72736 bytes,4416160 total

- age7:24264 bytes,4440424 total

- age8:7512 bytes,4447936 total

- age9:102184 bytes,4550120 total

- age 10:131752 bytes,4681872 total

- age 11:33544 bytes,4715416 total

- age 12:68504 bytes,4783920 total

- age 13:124048 bytes,4907968 total

- age 14:209808 bytes,5117776 total

- age 15:31840 bytes,5149616 total

- age 16:177448 bytes,5327064 total

- age 17:24176 bytes,5351240 total

:178162K->5225K(183808K), 0.1415803 secs] 427650K->254713K(436992K), 0.1418723 secs]

3872.574: [GC 3872.575: [DefNew

Desired survivor size 5505024 bytes, new threshold 14 (max 31)

- age1:4211472 bytes,4211472 total

- age2:167704 bytes,4379176 total

- age3:59912 bytes,4439088 total

- age4:99920 bytes,4539008 total

- age5:407376 bytes,4946384 total

- age6:106472 bytes,5052856 total

- age7:72736 bytes,5125592 total

- age8:24264 bytes,5149856 total

- age9:7512 bytes,5157368 total

- age 10:102184 bytes,5259552 total

- age 11:131752 bytes,5391304 total

- age 12:33544 bytes,5424848 total

- age 13:68504 bytes,5493352 total

- age 14:124048 bytes,5617400 total

- age 15:209808 bytes,5827208 total

- age 16:31840 bytes,5859048 total

- age 17:119096 bytes,5978144 total

- age 18:23808 bytes,6001952 total

:178281K->5861K(183808K), 0.1315884 secs] 427769K->255348K(436992K), 0.1319066 secs]

3877.527: [GC 3877.528: [DefNew:178917K->178917K(183808K), 0.0000662 secs]3877.528: [Tenured[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor125]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor51]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor75]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor67]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor16]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor71]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor81]

[Unloading class sun.reflect.GeneratedMethodAccessor31]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor139]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor18]

[Unloading class sun.reflect.GeneratedConstructorAccessor19]

[Unloading class sun.reflect.GeneratedConstructorAccessor55]

[Unloading class sun.reflect.GeneratedConstructorAccessor15]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor58]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor136]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor134]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor123]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor143]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor101]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor135]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor117]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor47]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor48]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor72]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor25]

[Unloading class sun.reflect.GeneratedConstructorAccessor22]

[Unloading class sun.reflect.GeneratedMethodAccessor22]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor99]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor28]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor33]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor108]

[Unloading class sun.reflect.GeneratedConstructorAccessor42]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor132]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor128]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor102]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor129]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor147]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor4]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor19]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor21]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor24]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor90]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor100]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor36]

[Unloading class sun.reflect.GeneratedConstructorAccessor61]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor145]

[Unloading class sun.reflect.GeneratedConstructorAccessor23]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor30]

[Unloading class sun.reflect.GeneratedMethodAccessor32]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor113]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor55]

[Unloading class sun.reflect.GeneratedMethodAccessor41]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor42]

[Unloading class sun.reflect.GeneratedConstructorAccessor48]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor41]

[Unloading class sun.reflect.GeneratedConstructorAccessor11]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor45]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor98]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor89]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor126]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor130]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor122]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor65]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor120]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor59]

[Unloading class sun.reflect.GeneratedConstructorAccessor49]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]

[Unloading class sun.reflect.GeneratedMethodAccessor24]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor127]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor97]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor93]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor20]

[Unloading class sun.reflect.GeneratedConstructorAccessor25]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor118]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor95]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor85]

[Unloading class sun.reflect.GeneratedMethodAccessor40]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor91]

[Unloading class sun.reflect.GeneratedMethodAccessor13]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor13]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor146]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor46]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor116]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor52]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor5]

[Unloading class sun.reflect.GeneratedMethodAccessor9]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor39]

[Unloading class sun.reflect.GeneratedMethodAccessor25]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor31]

[Unloading class sun.reflect.GeneratedMethodAccessor15]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor69]

[Unloading class sun.reflect.GeneratedConstructorAccessor53]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor66]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor56]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor92]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor68]

[Unloading class sun.reflect.GeneratedConstructorAccessor21]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor138]

[Unloading class sun.reflect.GeneratedConstructorAccessor63]

[Unloading class sun.reflect.GeneratedMethodAccessor39]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor15]

[Unloading class sun.reflect.GeneratedMethodAccessor6]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor111]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor96]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor64]

[Unloading class sun.reflect.GeneratedConstructorAccessor59]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor114]

[Unloading class sun.reflect.GeneratedConstructorAccessor58]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor141]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor107]

[Unloading class sun.reflect.GeneratedMethodAccessor14]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor54]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor26]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor3]

[Unloading class sun.reflect.GeneratedMethodAccessor43]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor106]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor140]

[Unloading class sun.reflect.GeneratedMethodAccessor44]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor49]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor78]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor73]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor79]

[Unloading class sun.reflect.GeneratedConstructorAccessor20]

[Unloading class sun.reflect.GeneratedMethodAccessor27]

[Unloading class sun.reflect.GeneratedMethodAccessor5]

[Unloading class sun.reflect.GeneratedConstructorAccessor60]

[Unloading class sun.reflect.GeneratedConstructorAccessor27]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor8]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor119]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor43]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor22]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor63]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor35]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor37]

[Unloading class sun.reflect.GeneratedMethodAccessor10]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor115]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor11]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor86]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor82]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor70]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor142]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor133]

[Unloading class sun.reflect.GeneratedConstructorAccessor46]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor40]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor84]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor121]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor94]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor53]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor74]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor88]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor110]

[Unloading class sun.reflect.GeneratedMethodAccessor23]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor44]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor7]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor57]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor80]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor76]

[Unloading class sun.reflect.GeneratedMethodAccessor12]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor12]

[Unloading class sun.reflect.GeneratedMethodAccessor42]

[Unloading class sun.reflect.GeneratedMethodAccessor33]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor83]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor62]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor50]

[Unloading class sun.reflect.GeneratedMethodAccessor26]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor144]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor109]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor38]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor124]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor61]

[Unloading class sun.reflect.GeneratedMethodAccessor7]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor77]

[Unloading class sun.reflect.GeneratedMethodAccessor29]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor103]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor60]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor112]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor137]

[Unloading class sun.reflect.GeneratedConstructorAccessor56]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor131]

:249487K->196086K(253184K), 2.5987253 secs] 428404K->196086K(436992K),2.6010559 secs]

3882.998: [GC 3882.998: [DefNew

Desired survivor size 5505024 bytes, new threshold 1 (max 31)

- age1:7461040 bytes,7461040 total

:173056K->7286K(183808K), 0.1877607 secs] 369142K->203372K(510624K), 0.1880744 secs]

3886.814: [GC 3886.814: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:5084112 bytes,5084112 total

:180342K->4964K(183808K), 0.1731598 secs] 376428K->204879K(510624K), 0.1734656 secs]

[19294 byte] By [jmkdeva] at [2007-10-3 11:15:13]
# 1

Looking at the age distribution of objects in your logs, it would

seem to me that concurrent GC may work well in your situation

in reducing the pause times seen for collecting the dead objects from

the old generation. When using concurrent GC, recall that class

unloading is not enabled by default so you will need to explicitly

enable it via -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled.

When using CMS, you might be best off not keeping the "medium-lived" objects

in the young generation beyond a single scavenge (i.e. a MaxTenuringThreshold

of 2).

As regards your question of what's in the heap, you might find

jmap -histo (or its historical precursor -XX:+PrintClassHistogram)

useful. For object connectivity and use, you might want to try a Java IDE

(with which latter my experiences are rather limited, but hopefully someone

on the forum might be able to recommend something suitable).

ramki_at_jdca at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 2

> Looking at the age distribution of objects in your

> logs, it would

> seem to me that concurrent GC may work well in your

> situation

> in reducing the pause times seen for collecting the

> dead objects from

> the old generation. When using concurrent GC, recall

> that class

> unloading is not enabled by default so you will need

> to explicitly

> enable it via -XX:+CMSClassUnloadingEnabled

> -XX:+CMSPermGenSweepingEnabled.

>

> When using CMS, you might be best off not keeping the

> "medium-lived" objects

> in the young generation beyond a single scavenge

> (i.e. a MaxTenuringThreshold

> of 2).

>

> As regards your question of what's in the heap, you

> might find

> jmap -histo (or its historical precursor

> -XX:+PrintClassHistogram)

> useful. For object connectivity and use, you might

> want to try a Java IDE

> (with which latter my experiences are rather limited,

> but hopefully someone

> on the forum might be able to recommend something

> suitable).

Try with eclipse TPTP project. It磗 powerful and its free.

david327a at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 3

You are right that reflection (and some other operations) generate code as classes with unique names. Those classes can be collected as the reflective access completes. However, they are not filling up the Java object heap, but a separate memory area called the "permanent" generation, misnamed, since, as you can see things there are not permanent.

Ramki is right that you might be a good candidate for CMS. Look at http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html for more details on choosing and tuning a collector.

The other thing to watch is that your live data size (e.g., the size of the heap after a full collection) is 196086KB, or 191MB. You have a 436992KB heap, which I'll call 427MB, of which 183808K, or 180MB is in the young generation. When you have promoted enough objects from the young generation to the old generation to make it impossible for us to know if a young generation will succeed (e.g., you don't have 180MB free in the old generation), we will perform the a full collection, as you have shown. If you made your young generation smaller, you should have more frequent, but shorter, young generation collections, and maybe stretch the time between full collections, if those are a problem. Your young generation collections are quite successful, scavenging through 180MB to discover that only 5MB of objects that are still in use.

PeterKesslera at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 4

Thanks for everyone's input on this issue. Yes, the full GC's are causing an issue. This app is very high volume with max response times in the neighborhood of a couple hundred milliseconds. With this said, it appears that a vendor package is performing the reflection and generating garbage at a very high rate. If best we could do to hold off on the major GC's still take around 3 seconds to run. This 3 second pause for a major GC violates an SLA with the customer. If we use concurrent GC, too much CPU resources are used for the concurrent processes and the request-per-second count drops to an unacceptable level. We do use TPTP for profiling code, but this is a vendor package so do not have the Java source code to perform a complete profiling exercise. What recommendations or best practices could I recommend to the vendor to remove the reflection since the impact of the reflection has such a negative impact on a high volume fast response time application?

jmkdeva at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 5

It's not the use of reflection, per se, that's the problem. Reflection generates short-lived classes and you see them unloaded at full collections. But that's normal. I can't say that the couple of hundred classes is "too many", especially since I don't know what the interval is between the full collections.

Full collection times with the serial collector are a function of the size of the heap, so since you seem to have only 200MB of live data (the size after a full collection), you might try shrinking your young generation and your overall heap to try to finish a full collection in less than your "couple of hundred milliseconds", but that's likely to be a stretch, since you can't realistically shrink the heap to less than 50% of its current size, and 50% of your current 2.6 second collection is still outside your SLA.

You might be asking for more than is possible with your hardware. I don't suppose I could sell you a beefier machine? I thought not. If you send more of a log, and the complete command line (at least all the tuning parameters), along with a description of the hardware to "HotSpotGC-Feedback (at) Sun.COM" we might be able to give you more tuning advice.

PeterKesslera at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 6

I guess it is hard to tell from the output that I posted. The appliucation response time for the app is in the neighborhood of 200ms @ 85 RPS (Requests Per Second), and therefore the reflection is creating the classes at a high rate. When we try lowering the young generation area then we seem to make GC run more often. It appears that we hit a point where our throughput drops too much and it looks like full or major GC is happening way too often. Is there a way to keep these reflection classes in a separate part of the heap so the GC algorithm doesn't have to deal with these reflection classes along with the fixed named classees that the Application uses. Will the GC always look at every loaded class during a full or major GC cycle?

jmkdeva at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 7

I wouldn't stress about the reflection classes. They are a normal part of the operation of the JVM and cleaning them up doesn't take that long. We have to look at all classes (during full collections) to see if any of them can be unloaded, so having the generated classes in a separate place from the application classes wouldn't help. (Except that we'd want to look at the generated classes, since they are much more likely to be collectable!)

You have a GC tuning problem. Look at the GC tuning guide for advice. E.g., http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html (Did you say which release of the JVM you are running?) If you need GC pauses to be consistently less than 200ms, then you probably want to look at the mostly-concurrent collector, -XX:+UseConcMarkSweepGC, but then you will be doing a different tuning exercise than the serial mark-sweep-compact collector, -XX:+UseSerialGC, you are using here.

If you post some logs run with -XX:+PrintGCTimeStamps -XX:+PrintGCDetails, we can give better advice.

PeterKesslera at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 8

Here is a portion of log files from 11/27/2006 and 11/28/2007 test. Here is the document that was used to tune this jvm. The concurrentGC model was tried back in November but the total throughput for the application dropped and the CPU consumption was too much.

We used the jdk 1.4 tuning guide instead of the one referenced above for java5. Many of the suggestions were tested, but we could not find the sweetspot for this app. Either the max pause is long or too much CPU is used to perfor GC. Any idea would be appreciated. I am hoping it won't cost too much CPU to maintain the throughput for this app.

the logs are too large to post here.

***********************************************************************

11/27/2006

2273.144: [GC 2273.144: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3396208 bytes,3396208 total

: 157778K->3316K(159168K), 0.2261145 secs] 407235K->253055K(486920K), 0.2264406 secs]

2276.309: [GC 2276.310: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3470296 bytes,3470296 total

: 157684K->3388K(159168K), 0.1262762 secs] 407423K->253323K(486920K), 0.1265662 secs]

2279.355: [GC 2279.355: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3316512 bytes,3316512 total

: 157756K->3238K(159168K), 0.1369122 secs] 407691K->253429K(486920K), 0.1372297 secs]

2282.385: [GC 2282.385: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3720832 bytes,3720832 total

: 157606K->3633K(159168K), 0.1278529 secs] 407797K->253992K(486920K), 0.1281862 secs]

2285.457: [GC 2285.458: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3322240 bytes,3322240 total

: 158001K->3244K(159168K), 0.1308660 secs] 408360K->253900K(486920K), 0.1317469 secs]

2288.603: [GC 2288.603: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3588232 bytes,3588232 total

: 157612K->3504K(159168K), 0.1387172 secs] 408268K->254309K(486920K), 0.1390618 secs]

2291.764: [GC 2291.764: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:4393800 bytes,4393800 total

: 157872K->4290K(159168K), 0.1342670 secs] 408677K->255295K(486920K), 0.1345969 secs]

2294.792: [GC 2294.792: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3414888 bytes,3414888 total

: 158658K->3334K(159168K), 0.1756551 secs] 409663K->254639K(486920K), 0.1760070 secs]

2297.884: [GC 2297.884: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3542544 bytes,3542544 total

: 157702K->3459K(159168K), 0.1850556 secs] 409007K->255075K(486920K), 0.1853680 secs]

2300.915: [GC 2300.915: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:4077600 bytes,4077600 total

: 157553K->3982K(159168K), 0.2801147 secs] 409168K->255901K(486920K), 0.2804382 secs]

2303.924: [GC 2303.924: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3341344 bytes,3341344 total

: 158350K->3263K(159168K), 0.2672433 secs] 410269K->255425K(486920K), 0.2676638 secs]

2306.996: [GC 2306.997: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3491320 bytes,3491320 total

: 157631K->3409K(159168K), 0.1561422 secs] 409793K->255738K(486920K), 0.1566506 secs]

2309.966: [GC 2309.967: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3475736 bytes,3475736 total

: 157777K->3394K(159168K), 0.1218246 secs] 410106K->256016K(486920K), 0.1221916 secs]

2313.085: [GC 2313.085: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3598728 bytes,3598728 total

: 157762K->3514K(159168K), 0.1696552 secs] 410384K->256326K(486920K), 0.1699282 secs]

2316.140: [GC 2316.141: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3305552 bytes,3305552 total

: 157882K->3228K(159168K), 0.1252838 secs] 410694K->256373K(486920K), 0.1255814 secs]

2319.236: [GC 2319.236: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3259800 bytes,3259800 total

: 157596K->3183K(159168K), 0.1461852 secs] 410741K->256607K(486920K), 0.1465003 secs]

2322.285: [GC 2322.285: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3441200 bytes,3441200 total

: 157551K->3360K(159168K), 0.1202034 secs] 410975K->257031K(486920K), 0.1206303 secs]

2325.319: [GC 2325.319: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3928024 bytes,3928024 total

: 157728K->3835K(159168K), 0.1309297 secs] 411399K->257905K(486920K), 0.1312920 secs]

2328.324: [GC 2328.324: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3123056 bytes,3123056 total

: 157867K->3049K(159168K), 0.1220317 secs] 411937K->257230K(486920K), 0.1223221 secs]

2331.474: [GC 2331.475: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3055352 bytes,3055352 total

: 157417K->2983K(159168K), 0.1145153 secs] 411598K->257319K(486920K), 0.1148209 secs]

2334.695: [GC 2334.695: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3136232 bytes,3136232 total

: 157216K->3062K(159168K), 0.1195330 secs] 411552K->257554K(486920K), 0.1198627 secs]

2337.745: [GC 2337.745: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3697880 bytes,3697880 total

: 157430K->3611K(159168K), 0.1825682 secs] 411922K->258220K(486920K), 0.1828969 secs]

2340.805: [GC 2340.805: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3388608 bytes,3388608 total

: 157468K->3309K(159168K), 0.1608910 secs] 412077K->258317K(486920K), 0.1611809 secs]

2343.628: [Full GC 2343.628: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3309696 bytes,3309696 total

: 157677K->3232K(159168K), 0.2826415 secs] 412685K->258397K(486920K), 0.2830491 secs]

2346.705: [GC 2346.705: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3428352 bytes,3428352 total

: 157600K->3348K(159168K), 0.1191751 secs] 412765K->258765K(486920K), 0.1194448 secs]

2349.625: [GC 2349.625: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3372536 bytes,3372536 total

: 157716K->3293K(159168K), 0.1223127 secs] 413133K->258894K(486920K), 0.1227043 secs]

2352.809: [GC 2352.810: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3509736 bytes,3509736 total

: 157661K->3427K(159168K), 0.1659027 secs] 413262K->259209K(486920K), 0.1662121 secs]

2355.788: [GC 2355.788: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3272304 bytes,3272304 total

: 157795K->3195K(159168K), 0.1225834 secs] 413577K->259205K(486920K), 0.1231079 secs]

2358.748: [GC 2358.748: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3522320 bytes,3522320 total

: 157563K->3439K(159168K), 0.1319910 secs] 413573K->259672K(486920K), 0.1323201 secs]

2361.927: [GC 2361.928: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3338224 bytes,3338224 total

: 157646K->3259K(159168K), 0.1889977 secs] 413879K->259891K(486920K), 0.1893130 secs]

2364.949: [GC 2364.949: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3898000 bytes,3898000 total

: 157627K->3806K(159168K), 0.1376028 secs] 414259K->260663K(486920K), 0.1379225 secs]

2367.936: [GC 2367.936: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3590936 bytes,3590936 total

: 157967K->3506K(159168K), 0.1474942 secs] 414825K->260487K(486920K), 0.1477834 secs]

2370.985: [GC 2370.985: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3298976 bytes,3298976 total

: 157837K->3221K(159168K), 0.1551363 secs] 414819K->260532K(486920K), 0.1555589 secs]

2373.936: [GC 2373.937: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3310200 bytes,3310200 total

: 157589K->3232K(159168K), 0.1209223 secs] 414900K->260635K(486920K), 0.1211858 secs]

2378.096: [GC 2378.096: [DefNew

Desired survivor size 2457600 bytes, new threshold 1 (max 31)

- age1:3608144 bytes,3608144 total

: 157076K->3523K(159168K), 0.1516114 secs] 414478K->261214K(486920K), 0.1519220 secs]

2381.254: [GC 2381.255: [DefNew: 157891K->157891K(159168K), 0.0000646 secs]2381.255: [Tenured: 257691K->222213K(327752K), 2.9630944 secs] 415582K->222213K(486920K), 2.9655149 secs]

2386.247: [GC 2386.247: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:5570560 bytes,5570560 total

: 174400K->5440K(179840K), 0.2635826 secs] 396613K->230791K(550200K), 0.2638810 secs]

2390.556: [GC 2390.556: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:5570560 bytes,5570560 total

: 179837K->5440K(179840K), 0.2418157 secs] 405188K->235434K(550200K), 0.2425728 secs]

2393.947: [GC 2393.947: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4598944 bytes,4598944 total

: 179840K->4491K(179840K), 0.2242683 secs] 409834K->235785K(550200K), 0.2245734 secs]

2397.270: [GC 2397.270: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4158536 bytes,4158536 total

: 178891K->4061K(179840K), 0.2879193 secs] 410185K->235783K(550200K), 0.2882168 secs]

2400.968: [GC 2400.969: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4322736 bytes,4322736 total

: 178461K->4221K(179840K), 0.1373616 secs] 410183K->236135K(550200K), 0.1376681 secs]

2404.381: [GC 2404.382: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4722464 bytes,4722464 total

: 178621K->4611K(179840K), 0.1648980 secs] 410535K->236696K(550200K), 0.1652233 secs]

2407.739: [GC 2407.740: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:5568800 bytes,5568800 total

: 179011K->5438K(179840K), 0.1649234 secs] 411096K->238035K(550200K), 0.1652505 secs]

2411.162: [GC 2411.163: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4873176 bytes,4873176 total

: 179838K->4758K(179840K), 0.1509994 secs] 412435K->237809K(550200K), 0.1513420 secs]

2414.636: [GC 2414.636: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4429264 bytes,4429264 total

: 179158K->4325K(179840K), 0.1486133 secs] 412209K->237570K(550200K), 0.1489135 secs]

2417.928: [GC 2417.928: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4213168 bytes,4213168 total

: 178725K->4114K(179840K), 0.2082007 secs] 411970K->237559K(550200K), 0.2085162 secs]

2421.443: [GC 2421.443: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4657040 bytes,4657040 total

: 178514K->4547K(179840K), 0.1344679 secs] 411959K->238132K(550200K), 0.1348028 secs]

2424.798: [GC 2424.799: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4267960 bytes,4267960 total

: 178947K->4167K(179840K), 0.1757591 secs] 412532K->238325K(550200K), 0.1762353 secs]

2428.134: [GC 2428.134: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4109696 bytes,4109696 total

: 178567K->4013K(179840K), 0.1868518 secs] 412725K->238357K(550200K), 0.1872873 secs]

2431.604: [GC 2431.604: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4867936 bytes,4867936 total

: 178413K->4753K(179840K), 0.1473605 secs] 412757K->239222K(550200K), 0.1478890 secs]

2434.971: [GC 2434.971: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4289728 bytes,4289728 total

: 179153K->4189K(179840K), 0.1600035 secs] 413622K->238835K(550200K), 0.1603369 secs]

2438.406: [GC 2438.407: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4473544 bytes,4473544 total

: 178296K->4368K(179840K), 0.1286354 secs] 412942K->239191K(550200K), 0.1289415 secs]

2441.995: [GC 2441.995: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:5140080 bytes,5140080 total

: 178768K->5019K(179840K), 0.1509930 secs] 413591K->240142K(550200K), 0.1513179 secs]

2445.387: [GC 2445.387: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4952336 bytes,4952336 total

: 178888K->4836K(179840K), 0.1515447 secs] 414011K->240163K(550200K), 0.1518447 secs]

2448.765: [GC 2448.765: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4281928 bytes,4281928 total

: 179236K->4181K(179840K), 0.1692290 secs] 414563K->239692K(550200K), 0.1696671 secs]

2452.154: [GC 2452.154: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4452808 bytes,4452808 total

: 178581K->4348K(179840K), 0.1851835 secs] 414092K->240000K(550200K), 0.1859546 secs]

2455.513: [GC 2455.513: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4164624 bytes,4164624 total

: 178748K->4067K(179840K), 0.1376570 secs] 414400K->239894K(550200K), 0.1379396 secs]

2458.906: [GC 2458.906: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4236840 bytes,4236840 total

: 178467K->4137K(179840K), 0.1824181 secs] 414294K->240146K(550200K), 0.1827838 secs]

2462.506: [GC 2462.506: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4190960 bytes,4190960 total

: 178537K->4092K(179840K), 0.1338775 secs] 414546K->240218K(550200K), 0.1341785 secs]

2466.749: [GC 2466.749: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4624376 bytes,4624376 total

: 178492K->4515K(179840K), 0.1768329 secs] 414618K->240848K(550200K), 0.1771411 secs]

2470.280: [GC 2470.280: [DefNew: 178915K->178915K(179840K), 0.0000634 secs]2470.281: [Tenured: 236332K->218154K(370360K), 2.7253819 secs] 415248K->218154K(550200K), 2.7258222 secs]

2476.597: [GC 2476.598: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:5570560 bytes,5570560 total

: 174400K->5440K(179840K), 0.1722734 secs] 392554K->225971K(550200K), 0.1725892 secs]

2479.670: [GC 2479.670: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4201392 bytes,4201392 total

: 179496K->4102K(179840K), 0.1641324 secs] 400028K->225295K(550200K), 0.1644163 secs]

2483.223: [GC 2483.224: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:4211504 bytes,4211504 total

: 178502K->4112K(179840K), 0.1435554 secs] 399695K->225495K(550200K), 0.1438742 secs]

2488.395: [GC 2488.396: [DefNew

Desired survivor size 2785280 bytes, new threshold 1 (max 31)

- age1:3957040 bytes,3957040 total

: 178512K->3864K(179840K), 0.1258626 secs] 399895K->225422K(550200K), 0.1261162 secs]

***********************************************************************

11/28/2006

5772.008: [GC 5772.008: [DefNew

Desired survivor size 5505024 bytes, new threshold 1 (max 31)

- age1:6051328 bytes,6051328 total

- age2:363960 bytes,6415288 total

- age3:102480 bytes,6517768 total

- age4:54848 bytes,6572616 total

- age5:44192 bytes,6616808 total

- age6:59008 bytes,6675816 total

- age7:51256 bytes,6727072 total

- age8:36088 bytes,6763160 total

- age9:61120 bytes,6824280 total

- age 10:48888 bytes,6873168 total

: 178194K->6712K(183808K), 0.1640235 secs] 418201K->246718K(526480K), 0.1643304 secs]

5774.900: [GC 5774.900: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4822248 bytes,4822248 total

: 179768K->4709K(183808K), 0.1586654 secs] 419774K->245798K(526480K), 0.1589921 secs]

5778.175: [GC 5778.175: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4548904 bytes,4548904 total

- age2:145656 bytes,4694560 total

: 177765K->4584K(183808K), 0.1662986 secs] 418854K->245674K(526480K), 0.1665766 secs]

5781.438: [GC 5781.439: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4397056 bytes,4397056 total

- age2:189784 bytes,4586840 total

- age3:104464 bytes,4691304 total

: 177640K->4581K(183808K), 0.1473127 secs] 418730K->245670K(526480K), 0.1476110 secs]

5784.921: [GC 5784.921: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4509072 bytes,4509072 total

- age2:220664 bytes,4729736 total

- age3:79648 bytes,4809384 total

- age4:51856 bytes,4861240 total

: 177513K->4747K(183808K), 0.1462706 secs] 418602K->245836K(526480K), 0.1466015 secs]

5788.276: [GC 5788.277: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4383288 bytes,4383288 total

- age2:291280 bytes,4674568 total

- age3:171280 bytes,4845848 total

- age4:19136 bytes,4864984 total

- age5:51560 bytes,4916544 total

: 177803K->4801K(183808K), 0.1391278 secs] 418892K->245890K(526480K), 0.1394590 secs]

5791.651: [GC 5791.652: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4521888 bytes,4521888 total

- age2:222160 bytes,4744048 total

- age3:267288 bytes,5011336 total

- age4:47496 bytes,5058832 total

- age5:19136 bytes,5077968 total

- age6:51560 bytes,5129528 total

: 177656K->5009K(183808K), 0.1306977 secs] 418746K->246098K(526480K), 0.1310546 secs]

5795.081: [GC 5795.082: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4321768 bytes,4321768 total

- age2:177872 bytes,4499640 total

- age3:113984 bytes,4613624 total

- age4:110192 bytes,4723816 total

- age5:47496 bytes,4771312 total

- age6:19136 bytes,4790448 total

- age7:51560 bytes,4842008 total

: 178065K->4728K(183808K), 0.1752914 secs] 419154K->245818K(526480K), 0.1758526 secs]

5798.353: [GC 5798.353: [DefNew

Desired survivor size 5505024 bytes, new threshold 6 (max 31)

- age1:4919032 bytes,4919032 total

- age2:284864 bytes,5203896 total

- age3:149320 bytes,5353216 total

- age4:49104 bytes,5402320 total

- age5:76280 bytes,5478600 total

- age6:35528 bytes,5514128 total

- age7:19136 bytes,5533264 total

- age8:51560 bytes,5584824 total

: 177784K->5453K(183808K), 0.1745859 secs] 418874K->246543K(526480K), 0.1749226 secs]

5801.647: [GC 5801.648: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4379408 bytes,4379408 total

- age2:245416 bytes,4624824 total

- age3:262888 bytes,4887712 total

- age4:24112 bytes,4911824 total

- age5:45080 bytes,4956904 total

- age6:76280 bytes,5033184 total

: 178509K->4915K(183808K), 0.1477479 secs] 419599K->246108K(526480K), 0.1480798 secs]

5805.824: [GC 5805.825: [DefNew

Desired survivor size 5505024 bytes, new threshold 6 (max 31)

- age1:4922824 bytes,4922824 total

- age2:193648 bytes,5116472 total

- age3:142832 bytes,5259304 total

- age4:205856 bytes,5465160 total

- age5:24112 bytes,5489272 total

- age6:44920 bytes,5534192 total

- age7:76280 bytes,5610472 total

: 177971K->5478K(183808K), 0.1397963 secs] 419164K->246672K(526480K), 0.1400685 secs]

5809.151: [GC 5809.152: [DefNew

Desired survivor size 5505024 bytes, new threshold 5 (max 31)

- age1:5092144 bytes,5092144 total

- age2:188480 bytes,5280624 total

- age3:85184 bytes,5365808 total

- age4:76016 bytes,5441824 total

- age5:205744 bytes,5647568 total

- age6:24112 bytes,5671680 total

: 178534K->5538K(183808K), 0.1656120 secs] 419728K->246850K(526480K), 0.1661111 secs]

5812.453: [GC 5812.454: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4441176 bytes,4441176 total

- age2:221896 bytes,4663072 total

- age3:147552 bytes,4810624 total

- age4:31152 bytes,4841776 total

- age5:76016 bytes,4917792 total

: 178108K->4802K(183808K), 0.1589130 secs] 419420K->246338K(526480K), 0.1592241 secs]

5815.794: [GC 5815.794: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4105000 bytes,4105000 total

- age2:190488 bytes,4295488 total

- age3:178472 bytes,4473960 total

- age4:23992 bytes,4497952 total

- age5:31152 bytes,4529104 total

- age6:76016 bytes,4605120 total

: 177858K->4497K(183808K), 0.1580382 secs] 419394K->246033K(526480K), 0.1583108 secs]

5819.185: [GC 5819.186: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4031216 bytes,4031216 total

- age2:207992 bytes,4239208 total

- age3:81536 bytes,4320744 total

- age4:121792 bytes,4442536 total

- age5:23920 bytes,4466456 total

- age6:19184 bytes,4485640 total

- age7:76016 bytes,4561656 total

: 177553K->4454K(183808K), 0.1680179 secs] 419089K->245990K(526480K), 0.1683593 secs]

5822.489: [GC 5822.490: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4396272 bytes,4396272 total

- age2:112856 bytes,4509128 total

- age3:173376 bytes,4682504 total

- age4:28968 bytes,4711472 total

- age5:121792 bytes,4833264 total

- age6:23920 bytes,4857184 total

- age7:19184 bytes,4876368 total

- age8:76016 bytes,4952384 total

: 177510K->4836K(183808K), 0.2301784 secs] 419046K->246372K(526480K), 0.2304825 secs]

5825.969: [GC 5825.969: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4838872 bytes,4838872 total

- age2:196568 bytes,5035440 total

- age3:84008 bytes,5119448 total

- age4:51616 bytes,5171064 total

- age5:28424 bytes,5199488 total

- age6:121176 bytes,5320664 total

- age7:23920 bytes,5344584 total

- age8:18240 bytes,5362824 total

- age9:75944 bytes,5438768 total

: 177892K->5311K(183808K), 0.1428722 secs] 419428K->246847K(526480K), 0.1431520 secs]

5829.179: [GC 5829.179: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4488664 bytes,4488664 total

- age2:352712 bytes,4841376 total

- age3:96624 bytes,4938000 total

- age4:27312 bytes,4965312 total

- age5:51616 bytes,5016928 total

- age6:28424 bytes,5045352 total

- age7:119232 bytes,5164584 total

- age8:23920 bytes,5188504 total

- age9:17264 bytes,5205768 total

- age 10:75944 bytes,5281712 total

: 177895K->5157K(183808K), 0.2082275 secs] 419431K->246694K(526480K), 0.2085530 secs]

5832.596: [GC 5832.596: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4349880 bytes,4349880 total

- age2:113960 bytes,4463840 total

- age3:300928 bytes,4764768 total

- age4:40296 bytes,4805064 total

- age5:27216 bytes,4832280 total

- age6:51616 bytes,4883896 total

- age7:28424 bytes,4912320 total

- age8:119232 bytes,5031552 total

- age9:23920 bytes,5055472 total

- age 10:17232 bytes,5072704 total

- age 11:75944 bytes,5148648 total

: 178213K->5027K(183808K), 0.1462829 secs] 419750K->246564K(526480K), 0.1465726 secs]

5835.893: [GC 5835.893: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4362824 bytes,4362824 total

- age2:195752 bytes,4558576 total

- age3:74072 bytes,4632648 total

- age4:177968 bytes,4810616 total

- age5:40296 bytes,4850912 total

- age6:27216 bytes,4878128 total

- age7:51616 bytes,4929744 total

- age8:28368 bytes,4958112 total

- age9:107320 bytes,5065432 total

- age 10:23920 bytes,5089352 total

- age 11:17232 bytes,5106584 total

- age 12:63976 bytes,5170560 total

: 178083K->5049K(183808K), 0.1433297 secs] 419620K->246585K(526480K), 0.1436493 secs]

5839.256: [GC 5839.256: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4184368 bytes,4184368 total

- age2:244800 bytes,4429168 total

- age3:88880 bytes,4518048 total

- age4:20216 bytes,4538264 total

- age5:177920 bytes,4716184 total

- age6:40296 bytes,4756480 total

- age7:27216 bytes,4783696 total

- age8:51616 bytes,4835312 total

- age9:28368 bytes,4863680 total

- age 10:107320 bytes,4971000 total

- age 11:23920 bytes,4994920 total

- age 12:17232 bytes,5012152 total

- age 13:63976 bytes,5076128 total

: 178105K->4957K(183808K), 0.1911896 secs] 419641K->246493K(526480K), 0.1932825 secs]

5842.665: [GC 5842.666: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4472144 bytes,4472144 total

- age2:151480 bytes,4623624 total

- age3:192800 bytes,4816424 total

- age4:36032 bytes,4852456 total

- age5:20216 bytes,4872672 total

- age6:177616 bytes,5050288 total

- age7:40296 bytes,5090584 total

- age8:27216 bytes,5117800 total

- age9:39648 bytes,5157448 total

- age 10:28368 bytes,5185816 total

- age 11:107320 bytes,5293136 total

- age 12:23920 bytes,5317056 total

- age 13:17232 bytes,5334288 total

- age 14:63976 bytes,5398264 total

: 178013K->5271K(183808K), 0.1327927 secs] 419549K->246807K(526480K), 0.1330712 secs]

5845.963: [GC 5845.963: [DefNew

Desired survivor size 5505024 bytes, new threshold 15 (max 31)

- age1:4671224 bytes,4671224 total

- age2:246392 bytes,4917616 total

- age3:111752 bytes,5029368 total

- age4:59840 bytes,5089208 total

- age5:35960 bytes,5125168 total

- age6:18264 bytes,5143432 total

- age7:73352 bytes,5216784 total

- age8:40296 bytes,5257080 total

- age9:27216 bytes,5284296 total

- age 10:39616 bytes,5323912 total

- age 11:28368 bytes,5352280 total

- age 12:107320 bytes,5459600 total

- age 13:23920 bytes,5483520 total

- age 14:17232 bytes,5500752 total

- age 15:63976 bytes,5564728 total

: 178327K->5434K(183808K), 0.2194104 secs] 419863K->246970K(526480K), 0.2197150 secs]

5849.254: [GC 5849.254: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4075344 bytes,4075344 total

- age2:422352 bytes,4497696 total

- age3:103760 bytes,4601456 total

- age4:56616 bytes,4658072 total

- age5:59672 bytes,4717744 total

- age6:35960 bytes,4753704 total

- age7:18264 bytes,4771968 total

- age8:39416 bytes,4811384 total

- age9:40296 bytes,4851680 total

- age 10:27216 bytes,4878896 total

- age 11:39616 bytes,4918512 total

- age 12:28200 bytes,4946712 total

- age 13:107320 bytes,5054032 total

- age 14:23848 bytes,5077880 total

- age 15:17232 bytes,5095112 total

: 178490K->4975K(183808K), 0.1670486 secs] 420026K->246574K(526480K), 0.1675405 secs]

5852.616: [GC 5852.617: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4114048 bytes,4114048 total

- age2:120128 bytes,4234176 total

- age3:389536 bytes,4623712 total

- age4:48016 bytes,4671728 total

- age5:44648 bytes,4716376 total

- age6:59672 bytes,4776048 total

- age7:35960 bytes,4812008 total

- age8:16320 bytes,4828328 total

- age9:39416 bytes,4867744 total

- age 10:40296 bytes,4908040 total

- age 11:27216 bytes,4935256 total

- age 12:39616 bytes,4974872 total

- age 13:28200 bytes,5003072 total

- age 14:107320 bytes,5110392 total

- age 15:23848 bytes,5134240 total

- age 16:17232 bytes,5151472 total

: 178031K->5030K(183808K), 0.1381689 secs] 419630K->246629K(526480K), 0.1385075 secs]

5856.573: [GC 5856.573: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4439696 bytes,4439696 total

- age2:165016 bytes,4604712 total

- age3:91680 bytes,4696392 total

- age4:162096 bytes,4858488 total

- age5:48016 bytes,4906504 total

- age6:44648 bytes,4951152 total

- age7:59672 bytes,5010824 total

- age8:35960 bytes,5046784 total

- age9:16320 bytes,5063104 total

- age 10:36432 bytes,5099536 total

- age 11:40296 bytes,5139832 total

- age 12:27216 bytes,5167048 total

- age 13:39616 bytes,5206664 total

- age 14:28200 bytes,5234864 total

- age 15:107320 bytes,5342184 total

- age 16:23848 bytes,5366032 total

- age 17:17232 bytes,5383264 total

: 178086K->5257K(183808K), 0.1437694 secs] 419685K->246855K(526480K), 0.1440939 secs]

5859.865: [GC 5859.866: [DefNew

Desired survivor size 5505024 bytes, new threshold 8 (max 31)

- age1:4819040 bytes,4819040 total

- age2:344432 bytes,5163472 total

- age3:72552 bytes,5236024 total

- age4:36768 bytes,5272792 total

- age5:128232 bytes,5401024 total

- age6:48016 bytes,5449040 total

- age7:32680 bytes,5481720 total

- age8:59168 bytes,5540888 total

- age9:35960 bytes,5576848 total

- age 10:15808 bytes,5592656 total

- age 11:36000 bytes,5628656 total

- age 12:40224 bytes,5668880 total

- age 13:27216 bytes,5696096 total

- age 14:39616 bytes,5735712 total

- age 15:28200 bytes,5763912 total

- age 16:95352 bytes,5859264 total

- age 17:23848 bytes,5883112 total

- age 18:17232 bytes,5900344 total

: 178313K->5762K(183808K), 0.1922176 secs] 419911K->247360K(526480K), 0.1925296 secs]

5863.189: [GC 5863.189: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4765992 bytes,4765992 total

- age2:137008 bytes,4903000 total

- age3:302768 bytes,5205768 total

- age4:19416 bytes,5225184 total

- age5:36768 bytes,5261952 total

- age6:128232 bytes,5390184 total

- age7:48016 bytes,5438200 total

- age8:32680 bytes,5470880 total

: 178818K->5342K(183808K), 0.1788654 secs] 420416K->247336K(526480K), 0.1791687 secs]

5866.623: [GC 5866.623: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4134248 bytes,4134248 total

- age2:212936 bytes,4347184 total

- age3:103200 bytes,4450384 total

- age4:182584 bytes,4632968 total

- age5:19416 bytes,4652384 total

- age6:36768 bytes,4689152 total

- age7:116264 bytes,4805416 total

- age8:48016 bytes,4853432 total

- age9:32680 bytes,4886112 total

: 178398K->4771K(183808K), 0.1600738 secs] 420392K->246765K(526480K), 0.1603875 secs]

5869.932: [GC 5869.932: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4217640 bytes,4217640 total

- age2:204240 bytes,4421880 total

- age3:109808 bytes,4531688 total

- age4:50656 bytes,4582344 total

- age5:182456 bytes,4764800 total

- age6:19416 bytes,4784216 total

- age7:36768 bytes,4820984 total

- age8:116264 bytes,4937248 total

- age9:48016 bytes,4985264 total

- age 10:32680 bytes,5017944 total

: 177827K->4900K(183808K), 0.1784827 secs] 419821K->246894K(526480K), 0.1796111 secs]

5873.286: [GC 5873.287: [DefNew

Desired survivor size 5505024 bytes, new threshold 2 (max 31)

- age1:5442344 bytes,5442344 total

- age2:184384 bytes,5626728 total

- age3:163704 bytes,5790432 total

- age4:54112 bytes,5844544 total

- age5:50656 bytes,5895200 total

- age6:182456 bytes,6077656 total

- age7:19416 bytes,6097072 total

- age8:36768 bytes,6133840 total

- age9:116264 bytes,6250104 total

- age 10:48016 bytes,6298120 total

- age 11:32680 bytes,6330800 total

: 177735K->6182K(183808K), 0.1688150 secs] 419729K->248176K(526480K), 0.1694435 secs]

5876.635: [GC 5876.635: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4428920 bytes,4428920 total

- age2:201816 bytes,4630736 total

: 179238K->4522K(183808K), 0.1359659 secs] 421232K->247200K(526480K), 0.1362973 secs]

5879.903: [GC 5879.903: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:3830536 bytes,3830536 total

- age2:187536 bytes,4018072 total

- age3:83320 bytes,4101392 total

: 177578K->4005K(183808K), 0.1731194 secs] 420256K->246683K(526480K), 0.1734423 secs]

5883.226: [GC 5883.226: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4342264 bytes,4342264 total

- age2:190416 bytes,4532680 total

- age3:151248 bytes,4683928 total

- age4:28632 bytes,4712560 total

: 177051K->4602K(183808K), 0.1805910 secs] 419730K->247280K(526480K), 0.1808881 secs]

5886.536: [GC 5886.536: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4226192 bytes,4226192 total

- age2:466480 bytes,4692672 total

- age3:161776 bytes,4854448 total

- age4:27688 bytes,4882136 total

- age5:28088 bytes,4910224 total

: 177658K->4795K(183808K), 0.1715613 secs] 420336K->247473K(526480K), 0.1718759 secs]

5890.617: [GC 5890.618: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4344496 bytes,4344496 total

- age2:321320 bytes,4665816 total

- age3:375752 bytes,5041568 total

- age4:87792 bytes,5129360 total

- age5:15720 bytes,5145080 total

- age6:28088 bytes,5173168 total

: 177851K->5051K(183808K), 0.1355771 secs] 420529K->247730K(526480K), 0.1359204 secs]

5893.978: [GC 5893.978: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4286320 bytes,4286320 total

- age2:275896 bytes,4562216 total

- age3:215640 bytes,4777856 total

- age4:318288 bytes,5096144 total

- age5:87776 bytes,5183920 total

- age6:15720 bytes,5199640 total

- age7:28088 bytes,5227728 total

: 178107K->5105K(183808K), 0.2050022 secs] 420786K->247783K(526480K), 0.2052954 secs]

5897.335: [GC 5897.335: [DefNew

Desired survivor size 5505024 bytes, new threshold 6 (max 31)

- age1:4567064 bytes,4567064 total

- age2:221440 bytes,4788504 total

- age3:228992 bytes,5017496 total

- age4:160464 bytes,5177960 total

- age5:302528 bytes,5480488 total

- age6:87776 bytes,5568264 total

- age7:15720 bytes,5583984 total

- age8:28088 bytes,5612072 total

: 178161K->5480K(183808K), 0.1601476 secs] 420839K->248159K(526480K), 0.1604941 secs]

5900.963: [GC 5900.963: [DefNew

Desired survivor size 5505024 bytes, new threshold 1 (max 31)

- age1:7403248 bytes,7403248 total

- age2:415080 bytes,7818328 total

- age3:182032 bytes,8000360 total

- age4:90992 bytes,8091352 total

- age5:160464 bytes,8251816 total

- age6:289592 bytes,8541408 total

: 178484K->8341K(183808K), 0.1632637 secs] 421163K->251136K(526480K), 0.1635886 secs]

5904.285: [GC 5904.285: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4528416 bytes,4528416 total

: 181397K->4422K(183808K), 0.2326822 secs] 424192K->248764K(526480K), 0.2332402 secs]

5907.531: [GC 5907.531: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:3902856 bytes,3902856 total

- age2:441512 bytes,4344368 total

: 177478K->4242K(183808K), 0.1377787 secs] 421820K->248585K(526480K), 0.1380795 secs]

5910.834: [GC 5910.834: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:3820640 bytes,3820640 total

- age2:439984 bytes,4260624 total

- age3:376016 bytes,4636640 total

: 177298K->4527K(183808K), 0.1797619 secs] 421641K->248870K(526480K), 0.1800454 secs]

5914.170: [GC 5914.171: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:3842248 bytes,3842248 total

- age2:374336 bytes,4216584 total

- age3:321536 bytes,4538120 total

- age4:317304 bytes,4855424 total

: 177583K->4741K(183808K), 0.1481430 secs] 421926K->249084K(526480K), 0.1484170 secs]

5917.517: [GC 5917.518: [DefNew

Desired survivor size 5505024 bytes, new threshold 31 (max 31)

- age1:4069840 bytes,4069840 total

- age2:275280 bytes,4345120 total

- age3:330904 bytes,4676024 total

- age4:264912 bytes,4940936 total

- age5:213944 bytes,5154880 total

: 177797K->5034K(183808K), 0.1517872 secs] 422140K->249376K(526480K), 0.1520954 secs]

6070.774: [GC 6070.775: [DefNew

Desired survivor size 5505024 bytes, new threshold 1 (max 31)

- age1:6047880 bytes,6047880 total

- age2:377160 bytes,6425040 total

- age3:237832 bytes,6662872 total

- age4:205576 bytes,6868448 total

- age5:262352 bytes,7130800 total

- age6:112856 bytes,7243656 total

: 178090K->7073K(183808K), 0.1004700 secs] 422432K->251416K(526480K), 0.1007202 secs]

jmkdeva at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 9

(0) Do you have a portion of the GC log that indicates CMS (concurrent

collection) activity, perhaps? Or if the logs are too big, which they likely

are, feel free to send them as an attachment to hotspotgc {dash} feedback

{at} sun {dot} com.

(1) Are you running on a single processor machine?

(2) (if so) Did you try the "incremental" version of CMS, aka iCMS?

(3) If you are running 1.4.2_12 (or better) or 5.0 (or better) you will be able

to run with -XX:+HandlePromotionFailure which will allow you to

circumvent (mostly) the pessimal full promotion guarantee. Currently

you promote about 3 MB per scavenge, and have a footprint of about

190 MB. With the shape of heap you are using you run into the pessimal

full promotion guarantee after roughly 20 scavenges or so. With +HandlePromotionFailure

you should be able to go for about 45 scavenges or so, decreasing the

frequency of full gc's to half that number. Depending on what you believe

is the life time of the medium-lived data in your application (which is

currently getting promoted), you may be able to increase your heap

and or reshape it to decrease the frequency of full collections even

further. But I guess that does not really solve the problem of always

avoiding these multi-second full gc's -- unless you somehow manage to

prevent all promotion into the old generation by means of using

very generously sized survivor spaces. The only other solution I can

see currently is to retry the iCMS tuning exercise.

If you have a Sun support contract (see developers.sun.com/services for

a list of offerings), you should be able to get more specific tuning assistance.

ramki_at_jdca at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 10

0) Pasted above

(1) No, the machines are multi-processor machines. One has 12 CPU's with 12 GB RAM and the other is 8CPU's with 12 GB RAM. The wierd thing that we have just identified is the Full GC's seem to take about a second longer when we compare the 8way to the 12 way. One thing I would like to point out is the Application consists of 8 JVM's running the different pieces of the app. So, the 8 way has 8 jvm's and the 12 way has 8 jvm's.

(2) I believe so but what settings would you recommend that I should try? (if so) Did you try the "incremental" version of CMS, aka iCMS?

(3) We are running JDK 1.4.2_11. I agree with you that this option may not take care of the multi-second GC's, but we will keep it in mind when we upgrade to 1.4.2_12.

We are planning to re-run the ICMS exercise and we are thinking about trying the AgressiveHeap option. Do you think the AgressiveHeap Option will provide anything useful being the 8 way machine has 8 JVM's?

Can you think of an area in the code where a small change would help reduce the excessive GC times?

I am trying to get the development team to install this application on another vendors HW and JVM to compare results. I am also looking into our Sun Support Contract if this issue cannot be resolved via the vendor and the forums.

jmkdeva at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...
# 11

I've looked at your (partial) logs.

The log from 11/27/2006 shows two full collections (and one that started out to be a full collection at 2343.628, but then decided that a young generation collection was enough). Those take 3 seconds each. I gather that's a problem. You are running in about 500MB of heap and have about 220MB of live data. You are allocating 175MB (your eden size) every 3 seconds or so. Young generation collections are pretty effective. You are only promoting 34MB in the 90 seconds between full collections, but because the old generation isn't that much larger than your live data size, we need to run a full collection as often as we do. Or rather, since your live data plus your young generation size isn't that much larger than your old generation size, we need to run full collections. See the tuning guide about the "Young Generation Guarantee". (We've fixed that in JDK-1.5.0.)

In the log from 11/28/2006 there aren't any full collections, so it's harder to say how those will turn out. You seem to be chugging along in half of your 500MB heap only doing young generation collections and not promoting anything to the old generation. It looks like you could run for a while that way, but it's hard to extrapolate from 5 minutes of data.

Oh! You have big machines, with lots of processors and memory. Putting more memory in the old generation should let you stretch out the time between old generation collections. Putting more memory in the young generation might let us collect more of your temporary objects without promoting them to the old generation. Don't commit more memory to the heap (or the whole JVM) than you have physical memory.

Usually promoting to the old generation is a loss, because stop-the-world full collections are a pain. But if the mostly-concurrent mark-sweep collector (CMS, enabled with -XX:+UseConcMarkSweepGC) can collect the garbage there in "spare" cycles, sometimes it's better to let not-quite-temporary data get promoted so CMS can deal with it.

You haven't said what your goals are, for CPU overhead for collections or GC induced pause times. Obviously you want no overhead and no pauses, but we aren't there yet. CMS should be able to give you shorter pauses, but will need to have more room for the old generation than you've given, for several reasons outlined in the tuning guide. I suspect that when you tried it before it was running continuously, which you probably wouldn't like. If you could move to a JDK-1.6.0 JVM we could offer you parallel old generation collection, which should do really well on your big boxes.

Are your several JVM's running many application threads in each JVM? Is the contention for processors going to be a problem? If you are considering CMS, remember that (especially in JDK-1.4.2) CMS uses a single collector thread, so a JVM running 12 active, garbage-producing threads can overwhelm the single collector thread. But for modest numbers of threads, on a machine with spare cycles, CMS may be your answer.

Do not use -XX:+AggressiveHeap. That option is for JVM's that are pretty much the only thing running on a machine, to encourage the JVM to be aggressive about resources. Not the thing to use on a machine with lots of JVM's. -XX:+AggressiveHeap also tunes a lot of other things in ways that you probably won't like. If you really feel like you need -XX:+AggressiveHeap, run it with -XX:+PrintCommandLineFlags (oops, that's only in JDK-1.5.0 and later) and pick the options you understand and like.

A Sun support contract would get more focused attention on your tuning exercise.

PeterKesslera at 2007-7-15 13:39:16 > top of Java-index,Java HotSpot Virtual Machine,Specifications...