SocketChannel created in CLOSE_WAIT and never cleaned up - REDUX

Hi,

I am having the same issue as is described in the following bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6215050

My jvm is :

java version "1.5.0_07"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-b03)

Java HotSpot(TM) Client VM (build 1.5.0_07-b03, mixed mode, sharing

running in a win2k3 environment

the bug is listed as closed/fixed, but it seems many people(including myself) are still having the same issue.

in a nutshell, i am accumulating connections on my tomcat servers that stay in a CLOSE_WAIT state...like this:

TCP172.19.1.130:16002172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16005172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16006172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16011172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16012172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16014172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16015172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16017172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16019172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16022172.19.1.130:8080CLOSE_WAIT

TCP172.19.1.130:16023172.19.1.130:8080CLOSE_WAIT

i get so many that network stops functioning and i have to restart tomcat.

Does anyone know if this is being worked on by SUN? Is there another bug report that they are doing their work on? Is there any work-around or fix?

please help, my production systems are not stable!

thanks

[1540 byte] By [Guadagno_Tonya] at [2007-10-3 2:53:16]
# 1

Tony, if you're really using SocketChannels in blocking mode with a read timeout as per the bug report, you can always avoid this problem by just using Sockets: or you could go over to a non-blocking model. I know these may be huge changes.

If you're not using SocketChannels in blocking mode with read timeouts there may be another explanation. (Does Tomcat really do this?)

ejpa at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...
# 2

Hi,

Thanks for you reply. I thought about what you said and thought it would a good idea to start over......Let me explain my problem.

For a log time now, my tomcat application will lock up. when it is locked I see MANY sockets in a close_wait state(see above). they go away when i restart tomcat. to diagnose the problem, i took the test application from the bug and i changed it to use the commons http client and ran it agains my tomcat servers

the result is that on BOTH the client running the test application and on the tomcat web server, i accumulate close_wait connections.

i guess i am not sure that it is the same problem as the bug, but it does seem to be a jvm issue since on the client, tomcat is not involved. the common layer is java.

here is the test app, please let me know what you think

package com.test;

/*

* $Author: $

* $Revision: $

* $LastChangedDate: $

* $Id: $

*/

import java.io.IOException;

import java.io.InputStream;

import java.net.SocketException;

import org.apache.commons.httpclient.HttpClient;

import org.apache.commons.httpclient.HttpException;

import org.apache.commons.httpclient.HttpMethod;

import org.apache.commons.httpclient.HttpStatus;

import org.apache.commons.httpclient.HttpVersion;

import org.apache.commons.httpclient.methods.GetMethod;

import org.apache.commons.httpclient.params.HttpClientParams;

import org.apache.commons.httpclient.params.HttpConnectionParams;

// Inspired by code from

// http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6215050

public class TestCloseHttp_ForPosting implements Runnable

{

public static final String SMTP_HOSTNAME = "somehost";

public void run()

{

HttpMethod getMeth = null;

String sErrorMsg = null;

String url = "http://" + SMTP_HOSTNAME + "/test/test.html";

try

{

HttpClient client = new HttpClient();

int iHttpTimeOut = 3000;

client.getParams().setParameter("http.protocol.version", HttpVersion.HTTP_1_1);

client.getParams().setParameter("http.socket.timeout", new Integer(iHttpTimeOut));

client.getParams().setParameter(HttpClientParams.ALLOW_CIRCULAR_REDIRECTS, false);

client.getParams().setParameter(HttpConnectionParams.SO_LINGER, 1);

getMeth = new GetMethod(url);

getMeth.setFollowRedirects(false);

int iHttpRetCode = client.executeMethod(getMeth);

if (iHttpRetCode != HttpStatus.SC_OK)

{

sErrorMsg = "client.executeMethod(getMeth)=" + iHttpRetCode;

sErrorMsg += "\n" + url;

System.err.println(sErrorMsg);

}

InputStream in = getMeth.getResponseBodyAsStream();

int iTest = 0;

byte bArr[] = new byte[1024 * 1];

while (iTest > -1)

{

iTest = in.read(bArr);

}

in.close();

long end = System.currentTimeMillis();

}

catch (HttpException e)

{

sErrorMsg = "HttpException: run";

sErrorMsg += "\n" + url;

System.err.println(sErrorMsg);

}

catch (SocketException e)

{

sErrorMsg = "SocketException: run";

sErrorMsg += "\n" + url;

System.err.println(sErrorMsg);

}

catch (IOException e)

{

sErrorMsg = "IOException: run";

sErrorMsg += "\n" + url;

System.err.println(sErrorMsg);

}

catch (Exception e)

{

sErrorMsg = "Exception: run";

sErrorMsg += "\n" + url;

System.err.println(sErrorMsg);

}

finally

{

getMeth.releaseConnection();

}

}

public static void main(String[] args)

{

TestCloseHttp_ForPosting test = new TestCloseHttp_ForPosting();

for (int i = 0; i < 15000; i++)

{

// this bug seems only to appear if different threads are reading the channels

Thread thread = new Thread(test);

thread.start();

try

{

thread.join();

}

catch (InterruptedException e)

{

}

}

System.err.println("Going to sleep.... run netstat -an | grep CLOSE_WAIT ");

while (true)

{

try

{

Thread.sleep(5000);

}

catch (InterruptedException e)

{

}

}

}

}

Guadagno_Tonya at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...
# 3

It is looking like the same bug. I forgot that on some platforms e.g. Solaris a timed read is performed using a selector internally.

The only comment I have on your code is that you're only guaranteed to call close() if there are no exceptions. Normally the close() should be in the finally {} block.

ejpa at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...
# 4

Yes, i forgot to put the close in a finally block, but i do that in my real code.

It does seem like this is the same bug.......

if there are any Sun people out there, please comment on this.....I would like to know if it is on someones radar and when it might be fixed.........

PLEASE!!!!!!!!

Guadagno_Tonya at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...
# 5

Now I don't think it is the same bug. Reasons:

(a) As far as I know Tomcat doesn't use SocketChannels at all. It does timed reads if you set a read timeout, otherwise I expect it will probably use infinite read timeouts.

(b) I doubt that the HttpClient uses SocketChannels either.

(c) You're getting it at both the client and the server. I'd like to know more about this. Are you doing timed reads at the client?

What platform are you on?

ejpa at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...
# 6

Hi All!

I know that I am jumping in between..but in my case also, the same thing is happening. Connections (AJP13) are been held by the apache and not releasing. As the last resort, I need to restart the Tomcat. I had tried all configuratuions, and no luck. I believe, that till max 200 connection, it will work and the moment , it is reached, it will lock itself. I had trid to put timeout too, but still the connections are on building on and on.

My platform are Linux/Apache2.0/Tomcat 5.5.9/JDK1.5.0_4

ash_jhonesa at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...
# 7

I'm having the same problem and I am indeed using non-blocking SocketChannels. I'm using JVM 1.5.0_06.

We have yet to reproduce in the lab but we are seeing this in the field.

Has anyone tried closing both the input and output: "socketChannel.socket().shutdownInput()" and "socketChannel.socket().shutdownOutput"?

Is there any definitive answer solution to this problem yet?

Thanks,

Karl

KarlHendersona at 2007-7-14 20:42:16 > top of Java-index,Core,Core APIs...