troubles analyzing heap dump using hat/jhat
We're running jdk 1.5 update 7 on a SuSE Linux 8 machine (kernel 2.4.21-309-smp). It's a pretty large JBoss application server (4.0.3) running an embedded tomcat (5.5) and Systinet web services server (6.5).We've started running out of memory (java.lang.OutOfMemoryError: Java heap space) on a weekly basis.We're currently running these vm params (after trying many different options, including none):
-server -Xms768m -Xmx1024m -XX:+Use
ParallelGC -XX:+AggressiveHeap -XX:PermSize=128m -XX:MaxPermSize=128m -verbosegc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+
PrintTenuringDistribution
The gc output shows that PSOldGen grows consistently throughout the life of the vm, eventually nearing capacity (99.9999%) and either running out of memory or doing full GCs constantly while the application hangs.
I'm assuming a leak(s) in our code and am trying to track it down by getting and analyzing heap dumps. I was able to get a heap and analyze a dump in our test environment (where we can't reproduce the problem, of couse). However, I have generated 4 heap dumps now in production (using jmap -heap:format=b when the vm was nearing capacity) and I have not been able to analyze the dump file with either jhat or hat (using: jhat -stack false -J-mx1024m heap.bin and hat.big -stack=false heap.bin). All attempts result in variations of:
Read heap sub-record type 168 at position 0x24566d96
java.io.IOException: Unrecognized heap dump sub-record type: 168
at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:485)
at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:222)
at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:79)
at com.sun.tools.hat.Main.main(Main.java:143)
I'd appreciate any suggestions on how to get a valid heap dump file or other methods to identify our leak.
Thanks,
Jacob
