Webserver 6.1sp7 and Java 1.5 - find_vma failed
When starting up the server, I get this warning message:
warning (31055): CORE3283: stderr: Java HotSpot(TM) Server VM warning: Can't detect initial thread stack location - find_vma failed
I encountered it about a month ago but was able to fix it. Now I can't find how I fixed it in my notes or related answers here or through google. Perhaps it fixed itself on a restart (which I have now tried several times).
My problem is that the webserver seems to have stopped responding to requests on ports 80 and 443. The admin gui on 8888 works fine and indicates that the webserver is up on those ports.
An excerpt from the error log shows (many of these):
failure (31055): HTTP3069: Error accepting connection (PR_PROC_DESC_TABLE_FULL_ERROR: file descriptor table full)
I think the warning and this failure might be related.
Of course, this is a production machine, so I need to get it back up asap.
I am using the default, out-of-box, settings for nearly everything. I have had this server running for 6 months prior with no problem, so am really confused about what happened suddenly in the middle of the night. 3:01am on Apr 15 is the last access log entry.
I'm running RHEL 3 up8
Any help would be appreciated! Thanks. :)
[1305 byte] By [
EWC-guya] at [2007-11-27 1:13:36]
# 1
> When starting up the server, I get this warning
> message:
>
> warning (31055): CORE3283: stderr: Java
> HotSpot(TM) Server VM warning: Can't detect initial
> thread stack location - find_vma failed
>
<b>i have seen this before and it comes from jvm and it is most likely harmless. </b>
> I encountered it about a month ago but was able to
> fix it. Now I can't find how I fixed it in my notes
> or related answers here or through google. Perhaps it
> fixed itself on a restart (which I have now tried
> several times).
>
> My problem is that the webserver seems to have
> stopped responding to requests on ports 80 and 443.
> The admin gui on 8888 works fine and indicates that
> the webserver is up on those ports.
>
> An excerpt from the error log shows (many of these):
>
> failure (31055): HTTP3069: Error accepting
> connection (PR_PROC_DESC_TABLE_FULL_ERROR: file
> descriptor table full)
>
this means , you will need to do something like
ulimit -n 8192
on your shell before you start the server.
this error happens because the server has used all the file descriptor for serving the incoming requests. by default, on some machine, the default file descriptor allocated for a application (around 1024) is not sufficient for high load servers. to avoid this issue, you will need to increase the file descriptor limit allocated for your web server , by doing 'ulimit -n'
> I think the warning and this failure might be
> related.
>
> Of course, this is a production machine, so I need to
> get it back up asap.
>
> I am using the default, out-of-box, settings for
> nearly everything. I have had this server running for
> 6 months prior with no problem, so am really confused
> about what happened suddenly in the middle of the
> night. 3:01am on Apr 15 is the last access log
> entry.
>
> I'm running RHEL 3 up8
>
> Any help would be appreciated! Thanks. :)
# 2
Well, I'm glad to hear that the two aren't related.
Thanks for your solution!
I found another resource that indicated my file descriptor problem. My solution may not be as granular as might be warranted, but here is what I did in RHEL3:
Edit file /etc/security/limits.conf, add:
*hardnofile8192
Edit file /etc/rc.d/init.d/sshd, start ( ) section, add:
ulimit -SHn 8192
I'm now watching the output of
more /proc/sys/fs/file-nr
to track actual usage.
Here's the question that develops from this, then. This webserver has been running without issue since October, seeing, as I imagine, about the same load (I don't yet have stats). The only thing that recently changed was the java version -- done for two reasons: 1) DST, 2) another software requirement. I moved up to jdk 1.5.0_09. I have yet to implement the new software, so I expect the server load to be similar to October. So, why the sudden maxing out of file handlers?
The base soft and hard max used to be 1024. I just set them to 8192. The true ceiling is evidently 155564. I guess we'll see what happens when the search spiders come in tonight.
