Memory Consumption Issue
I've run the below sample code on a number of different 1.5 JVMs and seen the same issue. When using jconsole to monitor the JVM, I see a slow perpetual consumption of memory, but it is not a memory leak. If I use the "Perform GC" option in jconsole, all of the memory consumed by the program since it originally started is finally released. Otherwise if I let the program run for hours/days, it just slowly eats up more and more memory. I don't understand how blocking a thread on a Socket.accept() is causing this.
Watching jconsole, when a normal GC occurs, about 99% of the memory consumed since the last GC is recovered. Rarely does it appear that all of the available memory is recovered. Slowly over time, while the thread is blocked on the Socket.accept(), the process consumes more and more memory.
Anyone know why this is occurring? What can be done to prevent it?
I have a multithreaded java service that never had any problems running on 1.4. When I ported the code over to 1.5 my hosting server ran out of memory while running the service. The hosting server is an ancient desktop PC and didn't have much memory to begin with, which is how I discovered the problem. The full blown code for the multithreaded service running under 1.5, has the same memory comsumption pattern as this skinnied downed example. I haven't tried 1.6 yet.
import java.io.*;
import java.net.*;
import java.util.*;
public final class Server implements Runnable {
public void run() {
try {
ServerSocket serverSocket=new ServerSocket(22,20); // Port 22, Connection queue size of 20
Socket newClientSocket=serverSocket.accept();
}
catch (IOException e) {
}
}
public static void main(String[] args) {
Server server=new Server();
Thread serverThread=new Thread(server);
serverThread.start();
}
}
[1915 byte] By [
JoeG2007a] at [2007-11-26 20:18:20]
Anyone have any insights?
This may or maynot help you but one thing that I immediately noticed about your code is that you are swallowing exceptions - never a good thing. At the very least you should be printing out the exception so you can see what/if there is a problem. Better yet would be to deal with the
I've removed a lot of code from the example in order to make it more readable. The exception catch block is not being executed when this problem is occurring. I appreciate the input, but I don't think this is causing the memory growth.
> I have a multithreaded java service that never had
> any problems running on 1.4. When I ported the code
> over to 1.5 my hosting server ran out of memory while
> running the service.
Just as with previous versions I am sure 1.5 takes more memory for the VM itself than previous versions did.
If you were only the border before then this would cause a OOM.
Are you claiming that the code sample (and only the code sample) produces a OOM?
The service never actually threw a JVM OOM error. If I leave the service running for about 2 days, the CPU is trashing because it appears (after looking at the task manager) that the OS is paging memory to and from disk. When I kill the service, the CPU and memory consumption returns back to normal. This is repeatable. It's happened 3-4 times and I've since disabled the service.
Looking into the problem I ran jconsole against my unmodified server code and saw this gradual memory consumption. I've distilled my code down to the above example code, which also exhibits the same memory issue. No, I haven't run the example code to the point that it generates a JVM OOM error.
I do think I'm going to run my server with jconsole monitoring it for a few DAYS and see what those numbers look like when the server starts to have problems.
Maybe I'm chasing a red herring.
Are you careful to eventually close every socket accept() returns, eh?
> The service never actually threw a JVM OOM error. If
> I leave the service running for about 2 days, the CPU
> is trashing because it appears (after looking at the
> task manager) that the OS is paging memory to and
> from disk. When I kill the service, the CPU and
> memory consumption returns back to normal. This is
> repeatable. It's happened 3-4 times and I've since
> disabled the service.
>
> Looking into the problem I ran jconsole against my
> unmodified server code and saw this gradual memory
> consumption. I've distilled my code down to the
> above example code, which also exhibits the same
> memory issue. No, I haven't run the example code to
> the point that it generates a JVM OOM error.
>
> I do think I'm going to run my server with jconsole
> monitoring it for a few DAYS and see what those
> numbers look like when the server starts to have
> problems.
>
Still isn't clear.....with the sample code that you presented here and ONLY that code, are you seeing thrashing?
The problem has been identified and resolved. If you don't want the long explanation of the analysis and problem solving process, the issue was caused by failing to close sockets that were opened.
The "problem" with memory growth seen in jconsole when running the skeleton example code I posted in my original post is not an issue. I never ran the process long enough to discover that after 8-10 hours in the case of the above example code, the GC does a deeper memory recovery. I'm sure that there is some trigger other than time that causes these deeper GCs to occur, I just don't know what it is. (I was running these tests on a laptop that I had to shut down at the end of the day)
Here's my test scenario that helped me to find the problem...
My junker PC is running Win2k on a P2 Celron 533Mhz (actually overclocked) with 384MB memory. I submit below some example Client/Server code with a missing socket.close() on the Server side. When I run this on my junker PC, once I get ~300 threads processed, I can kill the client code, the server goes idle blocking on a socket.accept(), but the CPU is left in a permanent state of thrashing and spiking. If I kill the Server the CPU goes back to idle. If I uncomment the socket.close() in the Server class, the CPU usage remains in the 1-2% range even after processing thousands of threads.
I also ran the same example code from below on a new nice dual core Intel processor running WinXP. While the problem does occur, it is very very subtle and doesn't trash the machine. I generated 10-20k unclosed threads on the server and the dual core CPU was repeatedly spiking from 0 to 2-4% on usage until I killed the Server code. Not nearly as noticeable (still bad code).
It appears to me that the reason I missed closing the socket in my service code was my choice of variable names. :( I closed the InputStreamReader I was using to read from the socket, but forgot to close the socket because the names of the two variables looked too similar.
Anyways, I appreciate the help. Sorry for the wild goose chase.
// Standard Java Packages
import java.net.*;
public final class Client {
public static void main(String[] args) {
while (true) {
try {
Socket socket=new Socket(InetAddress.getLocalHost(),22);
Thread.currentThread().sleep(1000);
socket.close();
}
catch (Exception e) {
}
}
}
}
// Standard Java Packages
import java.net.*;
public final class Server {
private static final class ClientConnection implements Runnable {
Socket socket=null;
ClientConnection(Socket socket) {
this.socket=socket;
}
public void run() {
try {
Thread.currentThread().sleep(500);
//Missing close statement that fixes the performance issue.
//socket.close();
}
catch (Exception e) {
}
}
}
public static void main(String[] args) {
Server server=new Server();
int threadCount=0;
try {
ServerSocket serverSocket=new ServerSocket(22,20); // Port 22, Connection queue size of 20
serverSocket.setReuseAddress(true);
while (true) {
System.out.println("Threads processed: "+threadCount);
// Block on socket read
Socket clientSocket=serverSocket.accept();
// Create a new thread to process this socket request
Thread serverThread=new Thread(new ClientConnection(clientSocket));
serverThread.start();
threadCount++;
}
}
catch (Exception e) {
}
}
}
