Java Memory Leaks - Growing Heap Space
I am having a few problems with a memory leak.
I have an application that listens on a socket for a trigger message before setting of a new thread to prcocess the request. This thread then instatiates an object that performs a web traversal search. The thread then dies when the search is complete.
However, after the thread has died the space on the heap used by the web traversal object remains on the heap. It is not removed even though it is now out of scope (declared and finishes in the run method) and the thread has finshed executing.
I have tried declaring the web traversal object to null after it has finished but this has no effect. Forcing garbage collection also does not improve the situation.
Anybody have any ideas that I have not thought of.
Thanks.
[805 byte] By [
C_J_a] at [2007-10-3 4:12:43]
Hi,Try to run the application through a memory profiler, and see from where the objects are referenced, and which ones that are growing. Kaj
kajbja at 2007-7-14 22:13:30 >
Does the object in question use any native resources such as database statements?
When you say that the object remains on the heap, do you just mean that when looking at the jconsole (or a profiler), the memory footprint does not go down?
Invoking System.gc() is not guaranteed to run ANY GC at all (although it usually does) and neither does it really mean that a full GC will be run, as the GC algorithm has to satisfy certain conditions when it runs. These conditions involve maximum amount of time in GC, percentage of time in GC etc etc. The only way that you can prove a memory leak is occurring is by ending up with an OutOfMemoryError as the GC will only throw this if it has no alternative.
If you're still sure that you have a memory leak then it is most likely to be something like:
1. Using a 3rd party piece of code that you don't fully understand (maybe you're not calling a close() method which releases resources).
2. You're adding something to a Collection and not removing it.
3. You're implementing the finalize() method (doesn't sound likely in your example)
4. You are holding onto Strings which were originally constructed by substring()-ing a much larger String.
> Invoking System.gc() is not guaranteed to run
> ANY GC at all (although it usually does) and neither
Jschell suggests that this is incorrect, as the APIs state:
When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all discarded objects.
> 3. You're implementing the finalize() method
> (doesn't sound likely in your example)
On its own implementing finalize() will not stop the GC from GCing.
Only if you do something bad in finalize().
--
@OP how are you checking that the space has not been returned?
mlka at 2007-7-14 22:13:30 >
> > Invoking System.gc() is not guaranteed to
> run
> > ANY GC at all (although it usually does) and
> neither
>
> Jschell suggests that this is incorrect, as the APIs
> state:
> When control returns from the method call, the
> Java Virtual Machine has made a best effort to
> reclaim space from all discarded objects.
I have also read some books which says that a full gc always is performed if a gc is started due to a call to gc(), but it isn't sure that a gc always is started:
From the javadoc for Runtime.gc():
"Runs the garbage collector. Calling this method suggests that the Java virtual machine expend effort toward recycling unused objects"
Kaj
kajbja at 2007-7-14 22:13:30 >
I do use many database statements.
C_J_a at 2007-7-14 22:13:30 >
download jvmstat from Sun and attach it to your server process. it'll give you a window into what's going on with memory. it won't tell you where the leak might be happening, but it will let you see how memory is migrated through the generations.
cleaning up database resources properly is very important. finalize is not the place to do it. finally blocks in local method scope is the right way.
%
if your server object is a singleton, see if it has an maps or other data structures that might be hanging onto objects and pinning them in memory. a weak reference map might be better in that case.
what tells you that heap space is growing? are you looking at the windows task manager? i don't believe that the memory shown in that tool for the Java process decreases when the GC runs. it's a ratchet - once the memory increases, it's always allocated to the jvm from the point of view of the OS, so it's not a good indicator into what's going on.
%
>When you say that the object remains on the heap, do you just mean that when looking at the jconsole >(or a profiler), the memory footprint does not go down?
I mean exactly that. I am using jProfiler. It shows that after the object has been used and has completed its task the footprint remains at the level when the object finishes and does not decrease. Subsequent uses of the object build upon this existing level. Eventually an OutOfMemoryException is Thrown.
I am using a 3rd party piece of code.
However, no such method exists with which to release the resources. This code creates a 'pool' of threads to perform the same job on many urls concurrently.
>2. You're adding something to a Collection and not removing it.
>
>3. You're implementing the finalize() method (doesn't sound likely in your example)
>
>4. You are holding onto Strings which were originally constructed by substring()-ing a much larger >String.
99% sure I am not doing any of these.
I am also trying to force garbage collection every 10 secs using System.gc()
Message was edited by:
C_J_
C_J_a at 2007-7-14 22:13:30 >
> >When you say that the object remains on the heap, do
> you just mean that when looking at the jconsole >(or
> a profiler), the memory footprint does not go down?
>
> I mean exactly that. I am using jProfiler. It shows
> that after the object has been used and has completed
> its task the footprint remains at the level when the
> object finishes and does not decrease. Subsequent
> uses of the object build upon this existing level.
> Eventually an OutOfMemoryException is Thrown.
You've got the right tool then.
OptimizeIt can go to the object/method level to look at what's happening. Can you do that with JProfiler?
> I am using a 3rd party piece of code.
> However, no such method exists with which to release
> the resources. This code creates a 'pool' of threads
> to perform the same job on many urls concurrently.
>
> >2. You're adding something to a Collection and not
> removing it.
> >
> >3. You're implementing the finalize() method
> (doesn't sound likely in your example)
> >
> >4. You are holding onto Strings which were
> originally constructed by substring()-ing a much
> larger >String.
>
> 99% sure I am not doing any of these.
>
> I am also trying to force garbage collection every 10
> secs using System.gc()
GC won't help at all if there are objections being created that are referred to by others. If the thread pool caches the ThreadLocal objects, and those create object references without cleaning them up, you can see how this might happen.
The ThreadPool should just be running your Runnables. I'd review all the Runnables that you use in the thread pool to make sure they aren't leaking.
%
> > >When you say that the object remains on the heap,
> do
> > you just mean that when looking at the jconsole
> >(or
> > a profiler), the memory footprint does not go
> down?
> >
> > I mean exactly that. I am using jProfiler. It
> shows
> > that after the object has been used and has
> completed
> > its task the footprint remains at the level when
> the
> > object finishes and does not decrease. Subsequent
> > uses of the object build upon this existing level.
> > Eventually an OutOfMemoryException is Thrown.
>
> You've got the right tool then.
>
> OptimizeIt can go to the object/method level to look
> at what's happening. Can you do that with
> JProfiler?
>
It will tell you what objects have been created but not how many times or where during the execution of the process. Or if they are still in the heap.
Does OptimizeIt go to this more detail?
Also I am using Mac OS 10.4 and I dont think this is platform is supported.
C_J_a at 2007-7-14 22:13:30 >
> It will tell you what objects have been created but
> not how many times or where during the execution of
> the process. Or if they are still in the heap.
>
> Does OptimizeIt go to this more detail?
I believe it does.
> Also I am using Mac OS 10.4 and I dont think this is
> platform is supported.
That I don't know. Windows and Unix for me.
%
> > OptimizeIt can go to the object/method level to
> look
> > at what's happening. Can you do that with
> > JProfiler?
> >
>
> It will tell you what objects have been created but
> not how many times or where during the execution of
> the process. Or if they are still in the heap.
Are you sure that JProbe can't do that? It was a long time ago since I used it, but I think it can show allocation graphs, and it looks like this article/link indicates that:
http://www.javaperformancetuning.com/tools/jprobe/index.shtml#MEMORY_DEBUGGING
Kaj
kajbja at 2007-7-14 22:13:30 >
> Jschell suggests that this is incorrect, as the APIs
> state:
> When control returns from the method call, the
> Java Virtual Machine has made a best effort to
> reclaim space from all discarded objects.
I'm not sure about this one. I know that jschell knows his/her stuff as they've been around on these forums for a long time but I do know the behaviour I've seen from JVMs in the 1.5.0_06 JDK in the last few months when I was tracking down a memory leak (which turned out not to be).
You can cause System.gc() through the jconsole, see the VM free up memory but not as much as it could do (and would allow a short time later). I believe that this is due to the latest algorithm which attempts to service certain targets. For example, it the percentage of time that it should spend in GC is 1%, then causing a System.gc() every second will not cause a full GC each time as it tries not to break the 1% target. I may be wrong about this but I don't think so, and it certainly fits in with the behaviour I've seen.
Maybe jschell him/herself can illuminate the situation.
> On its own implementing finalize() will not stop the GC from GCing.
> Only if you do something bad in finalize().
That's not true. It may be the case, that because of the processing time required in the finalize() method, that Java cannot get around to reclaiming the memory needed fast enough before an OutOfMemoryError occurs. I can prove this with the following code:
public class FinalizerOutOfMemoryTest {
private final static java.util.Random RANDOM = new java.util.Random();
private byte[] array_ = new byte[20 * 1024 * 1024];
public FinalizerOutOfmemoryTest() {
RANDOM.nextBytes(array_);
}
@Override
protected void finalize() throws Throwable {
Thread.sleep(1000);
super.finalize();
}
public static void main(String[] args) {
for (int i = 0; i < 1000; i++) {
FinalizerOutOfMemoryTest ft = new FinalizerOutOfMemoryTest();
}
}
}
On my PC this throws an OOME when the finalize method sleeps for more than 311ms. Less than that and it can just about proceed; I'd guess that this was also something to do with the GC targets.
..and you don't consider a sleep in the finalizer as something bad?
kajbja at 2007-7-21 10:26:07 >
> ..and you don't consider a sleep in the finalizer as something bad?
I take it this is a joke? Or you are being deliberatley obtuse? The finalize() method is there so that things like database resources etc. can be cleaned up. It's entirely possible that some object has resources which are expensive to clean up.
OK, so it's something of a pathological example to be creating large numbers of short-lived objects quickly (and which use expensive resources) which is why I said that this was unlikely but it's got nothing to do with anything illegitimate in the finalize method. I was merely using Thread.sleep() as a surrogate for some expensive operation.
> I take it this is a joke? Or you are being
> deliberatley obtuse? The finalize() method is
> there so that things like database resources etc. can
> be cleaned up. It's entirely possible that some
> object has resources which are expensive to clean
> up.
>
> OK, so it's something of a pathological
> example to be creating large numbers of
> short-lived objects quickly (and which use expensive
> resources) which is why I said that this was unlikely
> but it's got nothing to do with anything illegitimate
> in the finalize method. I was merely using
> Thread.sleep() as a surrogate for some
> expensive operation.
You should really avoid using the finalizer, and use close or dispose methods which you call when you are done using an object which is allocating native or "expensive" resources.
Kaj
kajbja at 2007-7-21 10:26:07 >
> The finalize() method is there so that > things like database resources etc.I'd say it is for when the developer forgets to call "close" on a native resource. A safety net. And then it is unlikely you will have 100s of native resources, each with long finilize
mlka at 2007-7-21 10:26:07 >
> You should really avoid using the finalizer, and use close or dispose methods
> which you call when you are done using an object which is allocating native or
> "expensive" resources.
Thanks for that. No, really, thanks. You see, I thought I'd just explained one of the reasons why it was dangerous to use the finalize() method to clear resources (that it could result in memory problems). But apparently I didn't and I needed you to tell me!
Why does everyone on these forums assume everyone else is so dumb?
> Thanks for that. No, really, thanks. You see, I
> thought I'd just explained one of the reasons why it
> was dangerous to use the finalize() method to
> clear resources (that it could result in
> memory problems). But apparently I didn't and I
> needed you to tell me!
It was rather a comment to your "pathological" example. There are times when finalizers are ok, and there are times when finalizers should be avoided, and I think that everyone agrees on the fact that you shouldn't perform lengthy operations in finalizers if you expect many objects to be allocated. I do also think that everyone agrees on close and dispose methods which could be called prior to the finalizer.
kajbja at 2007-7-21 10:26:07 >
> > Invoking System.gc() is not guaranteed to run
> > ANY GC at all (although it usually does) and
> neither
>
> Jschell suggests that this is incorrect, as the APIs
> state:
> When control returns from the method call, the
> Java Virtual Machine has made a best effort to
> reclaim space from all discarded objects.
>
Actually what I really point out is that the Sun VM supports hotloading of shared libraries and that the only way a shared library will reload is if the gc() is called twice. So the Sun VM gc() is doing something and doing it in a deterministic way. And has been doing that since 1.3.
And it must continue to do that for hotloading to work. I suspect any VM that works in a normal shared library environment would have to do something similar or the VM would not be able to support reloading shared libraries.
Not to mention that if you trace the code it certainly looks like it is doing something. There could of course be times when it doesn't do something though.
> I am having a few problems with a memory leak.
1. You verified that no child threads are created which never exit?
2. You are not loading a huge number of classes dynamically?
3. You are not calling 'intern' all over the place with dynamically created strings?
>
> I have tried declaring the web traversal object to
> null after it has finished but this has no effect.
> Forcing garbage collection also does not improve the
> situation.
That suggests it is a fairly major object which means actual references should be few. So trace every usage.
