shared memory: apache memory usage in solaris 10
Hi people, I have setup a project for the apache userID and set the new equivalent of shmmax for the user via projadd.In apache I crank up StartServers to 100 but the RAM is soon exhausted - apache appears not to use shared memory under solaris 10. Under the same version of apache in solaris 9 I can fire up 100 apache startservers with little RAM usage. Any ideas what can cause this / what else I need to do?Thanks!
# 1
Can you post some related information/data like the actual physical memory installed on the server, outputs of swap -l & projects -l, outputs of "vmstat 2 20" & "prstat -a" before and after running Apache server processes?
Are you not running multi-threaded Apache server?
>> In apache I crank up StartServers to 100 but the RAM is soon exhausted
I doubt it, unless you are running way too many processes than the system could handle. Most of the virtual memory will be shared between those apache processes. Please check 'SIZE' and 'RSS' columns of prstat output.
# 2
Hi there,
thanks for replying :-)
We have spent weeks and weeks looking at the situation hence my late reply, apologies.
But we now have a clear picture of the problem.
The issue boils down to two questions :
a) How or why does solaris choose to share memory between processes
from the same program invoked multiple times if that program has not
been *specifically* coded to use shared memory?
Simply: if we run pmap / ipcs we can see a shared memory reference
for our oracle database and ldap server. There is *no* entry for apache.
But the total memory usage is far far less than all the apache procs'
individual memory totted up (all 100 of them, in prstat.) So there is
some hidden sharing going on somewhere that solaris(2.9) is doing,
but not showing in pmap or ipcs.(virtually no swap is being used.)
b) Under solaris 10, each apache process takes up precisely the
memory reported in prstat - add up the 100 apache memory details
and you get the total RAM in use.crank up the number of procs any
more and you get out of memory errors so it looks like prstat is
pretty good here.The question is - why on solaris10 is apache not
'shared' but it is on solaris 9?We set up all the usual project details
for this user, (jn /etc/projects) but I'm guessing now that these project
tweaks where you explicitly set the shared memory for a user only take
effect for programs *explicitly* coded to use shared memory , e.g. the
oracle database, which correctly shows up a shared memory reference
in ipcs .
We can fire up thousands of apaches on the 2.9 system without
running out of memory - both machines have the same ram !
But the binary versions of apache are exactly the same, and
the config directives are identical.
please tell me that there is something really simple we have missed!
thanks!
# 3
> a) How or why does solaris choose to share memory
> between processes
> from the same program invoked multiple times
> if that program has not
> been *specifically* coded to use shared memory?
Take a look at 'pmap -x' output for a process.
Basically it depend on where the memory comes from. If it's a page loaded from disk (executable, shared library) then the page begins life shared among all programs using the same page. So a small program with lots of shared libraries mapped may have a large memory footprint but have most of it shared.
If the page is written to, then a new copy is created that is no longer shared. If the program requests memory (malloc()), then the heap is grown and it gathers more private (non-shared) page mappings.
> Simply: if we run pmap / ipcs we can see a
> shared memory reference
> for our oracle database and ldap server. There
> is *no* entry for apache.
> But the total memory usage is far far less than
> all the apache procs'
> individual memory totted up (all 100 of them, in
> prstat.) So there is
> some hidden sharing going on somewhere that
> solaris(2.9) is doing,
> but not showing in pmap or ipcs.(virtually
> no swap is being used.)
pmap -x should be showing you exactly which pages are shared and which are not.
> b) Under solaris 10, each apache process takes up
> precisely the
> memory reported in prstat - add up the 100
> apache memory details
> and you get the total RAM in use.crank up the
> number of procs any
> more and you get out of memory errors so it
> looks like prstat is
> pretty good here.The question is - why on
> solaris10 is apache not
> 'shared' but it is on solaris 9?We set up
> all the usual project details
> for this user, (jn /etc/projects) but I'm
> guessing now that these project
> tweaks where you explicitly set the shared
> memory for a user only take
> effect for programs *explicitly* coded to use
> shared memory , e.g. the
> oracle database, which correctly shows up a
> shared memory reference
>in ipcs .
> We can fire up thousands of apaches on the 2.9
> system without
> running out of memory - both machines have the
> same ram !
>
> But the binary versions of apache are exactly
> the same, and
>the config directives are identical.
> please tell me that there is something really
> simple we have missed!
On Solaris 10, do all the pages for one of the apache processes appear private? That would be really, really unusual.
--
Darren
# 4
Hi Darren, thanks very much for replying!
Your answer gives me the first hint of what is going on. And now I have to ask an ignorant question I'm afraid.How do I tell from pmap -xwhether a component of a process is private?So far, I have just been looking for the string 'shmid.'If I do pmap -x on an apache process, there is no shmid line. If I do pmap -x on an oracle database process e.g. smon, I do see this:
80000000 143896 143896- 143896 rwxsR[ ism shmid=0x500002f ]
- presumably because this is a program that explicitly codes a use of shared memory.
..as far as the'KbytesRSSAnon' columns go in pmap -x , the std doc pages do not help much, as far as I understand it , Kbytes is the total usage including swap, RSS is the resident set size, not including swap but INCLUDING anything shared, and anon I'm not sure about : "Memory not relating to any named object or file within the file system" ...
a thousand thanks!
best regards.
# 5
The columns should have that information:
23786: /bin/tcsh
AddressKbytes Resident Shared Private PermissionsMapped File
00010000296296296- read/exec tcsh
000680004040328 read/write/exectcsh
0007200052044838464 read/write/exec[ heap ]
FF100000688656656- read/exec libc.so.1
Resident is the portion in RAM. Shared are the pages that are available for sharing (although it may be that only one process currently has it mapped), while Private pages are not shared and are unique to the process.
--
Darren
# 6
>> We can fire up thousands of apaches on the 2.9 system without running out of memory - both machines have the same ram !
Which version of Solaris 10 are you running? Please post the output of "cat /etc/release".
[Solaris 10 Update 1 or later] Depending on the size of the heap and libraries, Solaris tries to use memory pages > 8K as appropriate (Large page OOB feature) - sometimes it may lead to some memory wastage.
The best thing to do in this scenario is to compare the outputs of "pmap -sx <apache_pid>" from Solaris 9 and 10 environments.
--
I think you are getting confused with the terms 'shared memory programming' and 'memory pages shared between processes'. Hope the following web page gives you some idea about shared memory, private memory etc.,
http://technopark02.blogspot.com/2005/12/solaris-estimating-process-memory.html
# 7
Hi giri04 and Darren, thanks for the replies, I will be attempting to digest these shortly, and promise to get back to you :-) I confirm it is on solaris 10 that we see the memory problem. Talk soon...
