Tuning Java App on an 8-core Sun Fire T2000 Server
I am deploying a java app on an 8-core (32 simultaneous threads) Sun Fire T2000 Server running Solaris 10 and have some questions about tuning the app and the JVM for this platform.
One thing that I have noticed is that the JVM starts up with 43 threads right out of the box. Is it normal for the JVM to do this (our JVM is build 1.5.0_09-b03)?
The flow of our application is this: read xml files from the file system (NAS), process data contained within the files, and then send filtered/corrected data in xml format to various WebsShere MQs (data is also archived to file system, NAS again).
The app is configured as two java processes both using a fixed thread pool (java.util.concurrent.Executors.newFixedThreadPool) of 14 worker threads. This configuration, 2 java processes with 14 threads each, has preformed better for us than one java process with 30 threads. Our problem is that even with two java processes running, vmstat reports about 60% user, 20% sys, and 20% idle cpu usage. We had previously worked on removing most of the blocking between the threads but I抦 thinking the high system usage from vmstat may mean that something is still there causing the threads to block/wait on each other. Is there anything else that high system usage in vmstat usually reflects in you java app?
In addition to the high sys usage in vmstat, prstat shows that many of our threads are sleeping. The prstat listing below shows that nearly half of the threads are sleeping 15-50% of the time. Our app pulls files from a directory and hands it off to a worker thread to process. For this test we loaded the input directory with plenty of files and therefore none of the worker threads should be sleeping.
Any suggestion on how many threads and/or how many java procs to run to best utilized the T2000?
Thanks,
Chris
> java -version
java version"1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
Java HotSpot(TM) Server VM (build 1.5.0_09-b03, mixed mode)
> prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
27871 user90 3.4 0.2 0.0 0.0 4.9 0.4 1.1 3K 1K 8K0 java/45
27859 user87 3.9 0.2 0.0 0.0 7.5 0.0 1.2 4K 749 10K0 java/47
27871 user87 3.7 0.1 0.0 0.0 6.6 1.5 0.8 3K 699 9K0 java/85
27871 user87 3.8 0.1 0.0 0.0 6.8 1.3 0.8 3K 650 9K0 java/44
27871 user87 3.8 0.1 0.0 0.0 6.5 1.6 0.8 3K 745 9K0 java/50
27859 user86 4.0 0.1 0.0 0.0 7.7 0.4 1.3 4K 661 10K0 java/45
27859 user86 4.2 0.2 0.0 0.0 8.1 0.8 0.9 4K 721 10K0 java/87
27871 user84 4.6 0.2 0.0 0.0 4.8 3.7 2.8 3K 839 10K0 java/67
27859 user83 4.8 0.1 0.0 0.0 7.8 3.2 1.3 4K 634 11K0 java/86
27859 user82 5.2 0.1 0.0 0.0 8.1 3.6 0.9 4K 593 12K0 java/50
27859 user82 3.8 0.1 0.0 0.0 7.9 5.2 0.8 3K 489 9K0 java/58
27859 user81 4.8 0.2 0.0 0.0 7.5 5.5 0.9 3K 978 10K1 java/85
27871 user72 7.6 0.2 0.0 0.0 2.5 12 4.9 3K 1K 15K0 java/65
27871 user71 7.7 0.1 0.0 0.0 2.7 13 5.0 3K 884 15K0 java/49
27871 user70 7.6 0.1 0.0 0.0 2.5 15 5.0 3K 612 15K0 java/52
27859 user70 7.5 0.2 0.0 0.0 7.4 13 1.6 4K 1K 15K1 java/60
27871 user68 8.1 0.2 0.0 0.0 4.4 17 1.9 2K 1K 15K0 java/47
27859 user68 7.9 0.2 0.0 0.0 7.6 15 1.1 3K 1K 15K0 java/52
27859 user68 8.2 0.1 0.0 0.0 7.7 15 1.0 4K 855 17K0 java/46
27871 user65 9.3 0.2 0.0 0.0 2.7 18 5.3 3K 1K 18K0 java/51
27871 user60 9.8 0.1 0.0 0.0 4.4 23 2.8 3K 740 19K0 java/66
27859 user58 9.9 0.1 0.0 0.0 7.2 23 1.4 3K 903 19K1 java/44
27871 user58 10 0.1 0.0 0.0 1.9 25 100 3K 993 19K0 java/46
27859 user52 12 0.2 0.0 0.0 5.8 28 2.3 3K 1K 23K0 java/51
27859 user48 12 0.1 0.0 0.0 7.1 32 1.0 3K 724 22K0 java/49
26917 user24 22 0.1 0.0 0.0 28 24 1.5 4K 230 54K0 amqrmppa/648674
27871 user11 35 0.0 0.0 0.0 0.0 54 0.3 300 163 19K0 java/43
27859 user10 35 0.0 0.0 0.0 1.5 50 2.9 427 175 17K0 java/43
27871 user24 18 0.1 0.0 0.0 2.1 51 5.3 3K 1K 32K0 java/84
26917 user15 16 0.0 0.0 0.0 21 45 1.8 4K 487 41K0 amqrmppa/648673
26917 user13 14 0.0 0.0 0.0 15 57 1.2 4K 253 36K0 amqrmppa/648672
20077 user13 14 0.0 0.0 0.0 15 56 1.8 4K 934 37K0 amqrmppa/17481
26910 user4.6 19 0.0 0.0 0.0 55 21 1.2 3K 128 7K0 amqzmuc0/3
26917 user11 12 0.1 0.0 0.0 16 59 1.3 3K 344 29K0 amqrmppa/648671
26917 user8.7 11 0.0 0.0 0.0 23 56 1.5 4K 183 30K0 amqrmppa/648665
26917 user8.2 11 0.1 0.0 0.0 21 59 1.4 4K 345 30K0 amqrmppa/648666
20077 user8.2 11 0.1 0.0 0.0 16 64 1.5 4K 888 28K0 amqrmppa/17477
28033 user9.0 7.6 0.0 0.0 0.0 83 0.0 0.7 1K 139 19K0 amqzlaa0_nd/162
28293 user8.7 4.8 0.0 0.0 0.0 86 0.0 0.8 2K 146 15K0 amqzlaa0_nd/17
27371 user8.9 4.5 0.1 0.0 0.0 86 0.0 0.9 2K 173 13K0 amqzlaa0_nd/3
Total: 148 processes, 506 lwps, load averages: 27.63, 23.88, 16.20
