Solaris Zones-Oracle memory issue
Hi,
We have installed three instances of Oracle on differenet zones across different servers. We followed all the best practices from sun & oracle. But our DBA say there is performance degradation on Sun servers as compare with the same installation on Intel servers. (He showed me some report take more time on Sun servers as compare on intel server!!!!)
My question is why the prstat reports that MEMORY is 100% on all three instances of oracle. ( Look at the end)
And why are oracle process sizes are so huge?
root@test# prstat -Z
PID USERNAME SIZERSS STATE PRI NICETIME CPU PROCESS/NLWP
24815 oracle3187M 3156M sleep5900:00:31 0.1% oracle/1
7536 oracle3187M 3159M sleep5900:00:00 0.0% oracle/1
7538 root3416K 3144K cpu05900:00:00 0.0% prstat/1
26072 noaccess 241M 155M sleep5900:03:00 0.0% java/27
24819 oracle3187M 3156M sleep5900:02:47 0.0% oracle/1
888 noaccess 217M 130M sleep5900:06:52 0.0% java/27
24823 oracle3186M 3154M sleep5900:02:40 0.0% oracle/1
24813 oracle3191M 3158M sleep5900:01:57 0.0% oracle/18
144 root7416K 6568K sleep5900:05:36 0.0% picld/13
24803 oracle3187M 3155M sleep5900:01:55 0.0% oracle/1
24821 oracle3188M 3161M sleep5900:00:25 0.0% oracle/1
307 root3664K 2424K sleep5900:00:00 0.0% snmpXdmid/2
302 root3088K 2040K sleep5900:00:00 0.0% dmispd/1
294 daemon2384K 1752K sleep60 -200:00:00 0.0% lockd/2
274 root2112K 1184K sleep5900:00:00 0.0% sf880drd/1
272 root2008K 1256K sleep5900:00:00 0.0% sac/1
268 daemon4392K 3296K sleep5900:00:00 0.0% nfsmapid/3
293 root1280K 944K sleep5900:00:00 0.0% utmpd/1
263 daemon2344K 1680K sleep60 -200:00:00 0.0% nfs4cbd/2
275 root5544K 4120K sleep5900:00:07 0.0% inetd/4
312 root2024K 1440K sleep5900:00:00 0.0% smcboot/1
292 root2392K 1728K sleep5900:00:00 0.0% ttymon/1
313 root2016K 1056K sleep5900:00:00 0.0% smcboot/1
26743 informix 1408K 1128K sleep5900:00:00 0.0% ksh/1
154 daemon4632K 2640K sleep5900:00:00 0.0% kcfd/3
151 root4072K 3072K sleep5900:00:21 0.0% nscd/26
167 root2296K 1192K sleep5900:00:25 0.0% in.mpathd/1
153 root5824K 3656K sleep5900:00:00 0.0% devfsadm/7
251 root2664K 1424K sleep5900:00:00 0.0% cron/1
218 root1240K 592K sleep5900:00:00 0.0% efdaemon/1
ZONEIDNPROC SIZERSS MEMORYTIME CPU ZONE
25959G59G100%0:12:55 0.1% kotcfleet
060 440M 273M0.4%0:14:31 0.1% global
1634 366M 230M0.4%0:03:31 0.0% informix
Any help or information will be appreciated
Message was edited by:
aakhan
Message was edited by:
aakhan
Message was edited by:
aakhan
[2733 byte] By [
aakhana] at [2007-11-27 8:24:42]
# 1
Performance is always a complex question -- there are many, many ways to affect it and, of course, we don't have all the details.
Given that your prstat shows 59gb of memory as resident for this zone, you must be on a pretty beefy box -- at least 64gb of RAM. If it is 64gb, I'd suspect that the oracle shared memory segments (eg. the SGA and company) are set too large -- you need to leave some memory for the rest of the processes on the machine. If you have more than 64gb, then maybe not.
What you really need to do is a through performance analysis of the machine to see where the bottle neck is (there's always a bottle neck -- even if its an acceptable one).
Unfortunately, this isn't an easily done thing -- it requires someone with both lots of experience in Solaris tuning and someone who knows your application reasonably well (this could be two people working together). If you don't have that talent in house you can hire it, or spend time reading up on it -- either way costs you, it's just that after gaining the skill yourself you'll have it for the next time (or for the next job :)
# 3
Hi MichaelMyers
This is only a V890 server with 16Gb ram. However I did some anaylisis with top & SEtool kit. Top reports,
load averages: 0.02, 0.02, 0.02;up 9+18:58:3308:17:23
60 processes: 59 sleeping, 1 on cpu
CPU states: 99.3% idle, 0.0% user, 0.7% kernel, 0.0% iowait, 0.0% swap
Memory: 16G phys mem, 10G free mem, 32G total swap, 32G free swap
PID USERNAME LWP PRI NICE SIZERES STATETIMECPU COMMAND
15988 root1 590 3032K 1768K cpu/00:00 0.17% top
24803 oracle1 590 3187M 3155M sleep3:37 0.02% oracle
24823 oracle1 590 3186M 3154M sleep5:03 0.00% oracle
24811 oracle18 590 3201M 3157M sleep0:36 0.00% oracle
24805 oracle1 590 3186M 3153M sleep0:21 0.00% oracle
3387 root25 590 4016K 2912K sleep0:28 0.00% nscd
3313 root15 590 8688K 6880K sleep0:10 0.00% svc.startd
3457 root4 590 5480K 4088K sleep0:11 0.00% inetd
3462 root1 590 1280K 880K sleep0:01 0.00% utmpd
24819 oracle1 590 3187M 3156M sleep5:24 0.00% oracle
24813 oracle18 590 3191M 3158M sleep3:41 0.00% oracle
24809 oracle258 590 3196M 3162M sleep1:12 0.00% oracle
24815 oracle1 590 3187M 3156M sleep0:59 0.00% oracle
24821 oracle1 590 3188M 3161M sleep0:48 0.00% oracle
3315 root13 590 9808K 8248K sleep0:16 0.00% svc.configd
I think what top reports is correct as compare to prtstat for whatever reasons. I also got similler results with SE toolkit. (But I still cannot figure out why orcale process are so huge)
What amount of performance over head should we expect from application due to zoning.
Bye
# 4
As Darren pointed out, shared memory makes things very difficult to figure out. I'm guessing the "real" answer is probably what top says plus a totaling of the sizes of the shared memory segments (which you can get from "ipcs -mob").
That said, you're focused on a performance metric from an application so you're probably best off doing profiling of that application to figure out where it's spending it's time. You can either do this from the application side (probably get your DBA's to do that, there are many tools such as Percise from Veritas) or from the system side (with something like dtrace).
The application side will yield the most useful information since it will report statistics directly that the application owners probably understand and can act on (eg. number of table scans, etc.).
Doing it from the system side will yield things like time spend in I/O, allocating memory, OS locks, etc. but these will require you and the application owner to work closely together to interpret to some type of useful changes which can be made to the application.
I might also suggest you move them outside the local zone to the global zone to remove that as a complicating factor. When/if you do, make sure you're using apples to apples -- same settings in the user profile, /etc/system, file system options, etc.
I wouldn't t expect zones to sap significant resources from the application -- maybe 5%.
