JVM Problems -- Client sessions being destroyed
We have been doing some stress testing of an internal app that we are
pushing through our Portal and have hit a bit of a performance wall. We
had around 900-950 clients with established sessions pretty equally
balanced between our 2 backend servers and for no explainable reason one
of the backends (the same one that is running the Profile services)
decided to fail and all the user sessions established to that one
machine are destroyed (this was consistent across all of our attempts).
Heres a snippet from the error log ...
[01/Aug/2001:18:41:00] info ( 3132): Aborting JVM
[01/Aug/2001:18:41:00] info ( 3132): Exiting JVM due to: jvm_abort ()
and jvm.exitOnAbort > 0
[01/Aug/2001:18:41:00] info ( 3132): JVM exit statistics:
AttachedThreads/Max=7/38, ActiveThreads/Max=13/38
This is the only error that appears in the logs.
I made a change to the jvm12.conf file to increase the available
heapsize for the JVM since originally we couldn't get past 350-450
client logins (I kept seeing a OutOfMemoryError being thrown to the
error logs). I don't think the changes made are the cause of the problem
since the second backend is handling the same number of connections fine
with the same changes made. Is it possible that the extra load from the
Profile server is the root of the problem? If so do you know of anyway
to split this service off that machine and onto a stand alone machine or
to even "turn off" handling of client sessions so that it will just do
the profile services? Or is there any more configuration changes that
can be made to avoid this problem allowing us to leave the current setup
the way it is? Or above all that does anyone at least have an
explanation for the above error message?
When the above error occurs the system remains active and clients
connected to the other backend are not effected but all the ones
connected to this machine die (which says to me that the Profile service
is not the one throwing the error). There is heavy CPU utilization and
high load (between 5 and 20), memory however does not seem to be an
issue. I've been holding off on the SP3 patch until it becomes a
*stable* patch release so we are still running with SP2 but if there is
anything other then the patch that can be done about this problem please
let me know.
Thanks,
Sean
--
Sean O'Grady
Information Technology - SS
Sheridan College
905-845-9430 x. 2166
sean.ogrady@sheridanc.on.ca
I can't remember which JVM version SP2 uses, but SP3 has 1.2.2_07 and
currently v1.2.2 is up to 1.2.2_09. First I'd check the OS has up to
date kernel/pthread patches to make sure this isn't the old garbage
collection/thread problem.
If you don't want to upgrade to SP3 yet, you could just try switching
the JVM for a later version and use the tuning suggestions in the SP3
release notes - Don't worry about changing the JVM heap size, it's
recommended anyway:
NameScopeDefault Recommended
jvm.min HeapSizejvm12.conf104857632768000
jvm.max HeapSizejvm12.conf16777216 805306368
For heavy accessed sites it is recommended to increase the max JVM heap
size to 768 MB in order to avoid a JVM abort problem due to a lack of
memory.
jvm.optionjvm12.conf-Xrunoii
JDK 1.2.2_07 provides better performance and scalability with the
following option: "-Xgenconfig:32m,32m,semispaces:32m,768m,markcompact
-Xoptimize"
Sean O'Grady wrote:
>
> We have been doing some stress testing of an internal app that we are
> pushing through our Portal and have hit a bit of a performance wall. We
> had around 900-950 clients with established sessions pretty equally
> balanced between our 2 backend servers and for no explainable reason one
> of the backends (the same one that is running the Profile services)
> decided to fail and all the user sessions established to that one
> machine are destroyed (this was consistent across all of our attempts).
> Heres a snippet from the error log ...
>
> [01/Aug/2001:18:41:00] info ( 3132): Aborting JVM
> [01/Aug/2001:18:41:00] info ( 3132): Exiting JVM due to: jvm_abort ()
> and jvm.exitOnAbort > 0
> [01/Aug/2001:18:41:00] info ( 3132): JVM exit statistics:
> AttachedThreads/Max=7/38, ActiveThreads/Max=13/38
>
> This is the only error that appears in the logs.
>
> I made a change to the jvm12.conf file to increase the available
> heapsize for the JVM since originally we couldn't get past 350-450
> client logins (I kept seeing a OutOfMemoryError being thrown to the
> error logs). I don't think the changes made are the cause of the problem
> since the second backend is handling the same number of connections fine
> with the same changes made. Is it possible that the extra load from the
> Profile server is the root of the problem? If so do you know of anyway
> to split this service off that machine and onto a stand alone machine or
> to even "turn off" handling of client sessions so that it will just do
> the profile services? Or is there any more configuration changes that
> can be made to avoid this problem allowing us to leave the current setup
> the way it is? Or above all that does anyone at least have an
> explanation for the above error message?
>
> When the above error occurs the system remains active and clients
> connected to the other backend are not effected but all the ones
> connected to this machine die (which says to me that the Profile service
> is not the one throwing the error). There is heavy CPU utilization and
> high load (between 5 and 20), memory however does not seem to be an
> issue. I've been holding off on the SP3 patch until it becomes a
> *stable* patch release so we are still running with SP2 but if there is
> anything other then the patch that can be done about this problem please
> let me know.
>
> Thanks,
> Sean
>
> --
> Sean O'Grady
> Information Technology - SS
> Sheridan College
> 905-845-9430 x. 2166
> sean.ogrady@sheridanc.on.ca
