iPlanet 6.1 Web server hanging up.

Hi,

I am facing the following problem:

The iplanet 6.1 webserver stops responding and when the users tries to access the site the following message shows up in the browser:

"The Page Cannot Be Displayed MSG HTTP 500 Internal Server Error"

And when i checked in the server log file I found the following error was logged before the server stopped responding:

java.io.IOException: WEB8004: Error flushing the output stream

at com.iplanet.ias.web.connector.nsapi.NSAPIConnector.flush(NSAPIConnector.java:74 3)

at com.iplanet.ias.web.connector.nsapi.NSAPIResponseStream.flush(NSAPIResponseStre am.java:97)

at com.iplanet.ias.web.connector.nsapi.NSAPIResponse.flushBuffer(NSAPIResponse.jav a:131)

at org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:21 9)

at org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:227)

at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:823)

at _jsps._misc._error_jsp._jspService(_error_jsp.java:87)

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:107)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:908)

at com.iplanet.ias.web.jsp.JspServlet$JspServletWrapper.service(JspServlet.java:68 7)

at com.iplanet.ias.web.jsp.JspServlet.serviceJspFile(JspServlet.java:459)

at com.iplanet.ias.web.jsp.JspServlet.service(JspServlet.java:375)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:908)

at org.apache.catalina.core.StandardWrapperValve.invokeServletService(StandardWrap perValve.java:771)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java: 322)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java: 218)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:209)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)

at com.iplanet.ias.web.connector.nsapi.NSAPIProcessor.process(NSAPIProcessor.java: 157)

at com.iplanet.ias.web.WebContainer.service(WebContainer.java:579)

Please let me know if any body has any idea about this problem and its cause. The remedy i am using as of now is restarting the web server.

[2483 byte] By [Rajarsia] at [2007-11-27 7:50:38]
# 1
>The iplanet 6.1 webserver stops responding Do you mean, after this error, webserver stop responding at all? DoesWebserver not respond to html pages after the error?
Basantka at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 2
Yes. It stops responding to any type of request though the web server process is running and we need to restart the web server to get rid of the situation.
Rajarsia at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 3
As another thread mentioned, this exception should not be a reason for the server hung. Please provide more details about your server, e.g version(service level)? configuration files and related logs.Our current 6.1release is 6.1SP7.
wyb2005a at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 4

"java.io.IOException: WEB8004: Error flushing the output stream"

This cannot be a reason for the server to be hanging. This exception means that client would have closed the connection before the response was sent to it.

Please paste the contents of your magnus.conf file. Maybe you are keeping the StackSize and rqThrottle to unreasonably high value because of which the server is hanging.

floyda at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 5

The details of the magnus.conf file is as following:

- -

#

# The NetsiteRoot, ServerName, and ServerID directives are DEPRECATED.

# They will not be supported in future releases of the Web Server.

NetsiteRoot /opt/iplanet61

ServerName sun86.domain.com

ServerID https-sun86.domain.com

#

RqThrottle 128

DNS off

Init fn="load-modules" funcs="ct-nsapi-init,ct-child-init,ct-pre-process,ct-post-process,ct-nsapi-user ,ct-auth,ct-check-redirect,ct-

nsapi-check-auth,ct-ssi-auth" shlib="/opt/rsa/ctrust/agent/agent-sjsw61-46/lib/libct_sjsws6_agent.so"

Init fn="ct-nsapi-init" CTAgentRoot="/opt/rsa/ctrust/agent/agent-sjsw61-46/webservers/sun86.domain.com"

Init fn="ct-child-init" LateInit=yes

Security off

Init fn="load-modules" funcs="wl_proxy,wl_init"\

shlib=/opt/iplanet61/plugins/libproxy.so

Init fn="wl_init"

PidLog /opt/iplanet61/https-sun86.domain.com/logs/pid

User webserve

StackSize 131072

TempDir /tmp/https-sun86.domain.com-7902a247

Init fn="load-modules" shlib="/opt/iplanet61/bin/https/lib/libj2eeplugin.so" shlib_flags="(global|now)"

Init fn="stats-init" profiling="on"

Init fn="flex-init" access="$accesslog" format.access="%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] \"%Req->reqpb.clf-reques

t%\" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% \"%Req->headers.referer%\" \"%Req->headers.user-agent%\" \"%Req->header

s.cookie%\" \"%vsid%\""

-

Rajarsia at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 6

From the config file, I see two plugins there. You use Weblogic proxy plugin and a RSA agent plugin. Did you test if you remove those plugins the problem is reporducible or not? The problem most likely come from plugins. You may get the dump from hanging instance and provide the stacktrace to get more help.

wyb2005a at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 7
Can you please give me the details about how to get the dump from hanging instance becaus in the log file I am not getting much details.
Rajarsia at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 8
kill -SIGABRT <pid> This will dump a core.
floyda at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 9
Can you please elaborate on this... the command that you have given is not working
Rajarsia at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...
# 10
use kill -6 <pid> or even better, use gcore <pid> to generate core file.
scandya at 2007-7-12 19:31:40 > top of Java-index,Web & Directory Servers,Web Servers...