How-to determine objects in the PS Old Generation space
Hi,
I am having issues with my PS Old Generation filling up and never gc'ing its objects. Obviously, I have some references to objects that shouldn't be. I would like to be able to look into the Old Generation heap space and find out what objects are in there to give me a clue as to what objects are hanging around. Does anyone know of such a tool, link references, or code I can hack to take a peek into the areas of the heap? The application is running under 1.5 JVM on a Solaris box.
If have used profilers such as Optimizeit and looked into JConsole using jmx, but I do not see in these tools a facility that will let me know what objects are in what area of the heap space. Perhaps these tools do what I need but I have missed it. If so, let me know that, too. This application is huge so it is not obvious where to begin to look for the leaks.
Thanks for any help you can give.
[911 byte] By [
garnettka] at [2007-10-3 5:53:26]
Hi,
Did you try 'jmap' ?
Usage: jmap [option] <pid>
(to connect to a live java process)
or jmap [option] <executable> <core>
(to connect to a core file)
or jmap [option] [server_id@]<remote server IP or hostname>
(to connect to a remote debug server)
where option must be one of:
<no option> to print same info as Solaris pmap
-heapto print java heap summary
-heap:format=bto dump java heap in hprof binary format
-histoto print histogram of java object heap
-permstatto print classloader-wise perm gen statistics
-h | -help to print this help message
jmap was already there in 1.5 - but if you can run your app on jdk 6 you
may find some additional goodies...
http://blogs.sun.com/alanb/entry/heap_dumps_are_back_with
http://blogs.sun.com/sundararajan/entry/observability_highlights_of_mustang
Or maybe DTrace can also help you there....
http://blogs.sun.com/sundararajan/entry/4_ways_to_view_java
[ try to query 'jmap site:blogs.sun.com' on www.google.com ]
-- daniel
http://blogs.sun.com/jmxetc
Thanks for the reply, Daniel.
I did use jmap and the histo option produced some revealing results. I was expecting a breakdown b/n New Gen heap and Old Gen, which jmap didn't reveal, however, after using jmx and JConsole under 1.5 I realized that the New Gen heap is very short lived (2-3 secs), so jmap -histo IS showing the Old Gen space.
FWIW in case others are having issues...
I have since resolved my problem, which was that the default GC collector on the 1.5 64-bit JVM was not ever doing a Full GC, which it is supposed to do at 68% of the Old Gen heap space, and the Old Gen space would just keep filling up until it hit capacity and choked itself out. I have since changed to using the Concurrent Mark Sweep collector for Old Gen cleanup and the ParNew for New Gen and the app is working great. The GC cleans up when it's supposed to and JConsole reveals that the application never did have leaks. The problem was in the JVM all along. Not sure if there is a bug reported on this anywhere, but it was definitely the issue.
Caveat: After changing collectors I ran jmap against the instance and it locked up on me and required a restart, so instead I enabled remote monitoring and used JConsole, which ended up being a blessing.
JConsole/jmx is a great improvement to the platform and was instrumental in figuring out this issue. My only wish list now, is if JConsole could display an instance count on the classes page. This would give one an idea where one could improve the application from an object creation/heap usage standpoint and it would also provide clues as to where an application might be having issues, if there are an unexpected high number of instances of a certain class. This would especially be handy right after a full GC has been completed.
For an idea of sizing for those interested, this app has around 30 - 45 concurrent, 600 - 1100 simultaneous (60 minute window) users. 10,000+ logins/day and runs on a 4-CPU, 8GB Sun Fire V440 Server. 3GB was allocated to the JVM before the fix and it crashed 2x/day. Now I have reduced the heap footprint to 2GB, which does a full GC every 30-40 minutes and doesn't crash.
thanks again,
Kevin
