Memory and swap problem

I have a question regarding memory and swap calculations in Solaris.

I have box that has 2GB of RAM and 3GB of swap.

I use ps command to check how much memory each program consumes. When I sum results I get numbers those are much smaller than memory and swap sizes: sum of the total size of the process in virtual memory - 866 664KB; sum of the resident set size of the process - 656 552KB.

At the same time, when I'm checking free memory space by vmstat, I see only 35 840KB of free memory space out of 2 GB and 17 688RB of free swap space out of 3GB.

When I try to run programs, in some cases, I get messages that there is not enough memory. Something like this:

can't mmap anonymous memory: Resource temporarily unavailable

not enough memory: Resource temporarily unavailable

Not enough space

How can I find out why I run out of swap and memory?

Have anyone faced the same situations and what could be done to get back "lost" hundreds of megabytes without box rebooting?

What could be the reason of such situation?

After the box reboot, "lost" megabytes "returned" but after some time the system again run out of memory and swap.

Outputs of some commands are below.

bash-2.05$ prtconf |head -2

System Configuration: Sun Microsystems sun4u

Memory size: 2048 Megabytes

bash-2.05$ swap -l

swapfile dev swaplo blocksfree

/dev/md/dsk/d385,316 40994248464

/swap/do_not_delete_swap_file ?,?16 2097136 166640

bash-2.05$ swap -s

total: 4604496k bytes allocated + 99648k reserved = [b]4704144k used[/b], 17768k available

bash-2.05$ vmstat 1 2

kthrmemorypagedisk faultscpu

r b wswap free re mf pi po fr de sr m0 m1 m3 m4insycs us sy id

0 0 0 2078664 646232 6 11 11 8 9 0 2 0 0 0 0 517 298 206 2 0 98

0 0 0 [b]17688 35840[/b]07 0 0 0 0 0 0 0 0 0 51057 119 0 0 100

bash-2.05$ ps -eo user,pid,vsz,rss,s,etime,args

USERPID VSZ RSS SELAPSED COMMAND

root000 T 8-23:32:55 sched

root1 1272 176 S 8-23:32:54 /etc/init -

root200 S 8-23:32:54 pageout

root300 S 8-23:32:54 fsflush

root263 1824 856 S 8-23:31:38 /usr/lib/saf/sac -t 300

root257 2712 1288 S 8-23:31:38 /usr/sbin/vold

root60 2336 752 S 8-23:32:48 /usr/lib/sysevent/syseventd

root68 2904 808 S 8-23:32:47 /usr/lib/picl/picld

root170 3744 1680 S 8-23:32:41 /usr/sbin/syslogd

root266 1832 880 S 8-23:31:37 /usr/lib/saf/ttymon

root148 2008 672 S 8-23:32:43 /usr/sbin/inetd -s

root174 2328 1112 S 8-23:32:41 /usr/sbin/cron

root190 3296 1904 S 8-23:32:40 /usr/sbin/nscd

root213 1776 536 S 8-23:32:40 /usr/sadm/lib/smc/bin/smcboot

root198 1096 616 S 8-23:32:40 /usr/lib/utmpd

root212 1776 848 S 8-23:32:40 /usr/sadm/lib/smc/bin/smcboot

root214 1776 536 S 8-23:32:40 /usr/sadm/lib/smc/bin/smcboot

root235 3096 864 S 8-23:31:39 /usr/sbin/mdmonitord

root264 1832 792 S 8-23:31:38 /usr/lib/saf/ttymon -g -h -p am-t-csmr1 console login: -T sun -d /dev/console

root268 3200 1216 S 8-23:31:37 /usr/lib/ssh/sshd

appuser 3896 810256 631360 S13:09:24 /export/home/appuser/j2sdk1.4.2_14/bin/java -server -Xms512m -Xmx1536m -XX:NewS

appuser 4218 2664 1968 S28:50 -bash

appuser 4230 1136 848 O0:-01 ps -eo user,pid,vsz,rss,s,etime,args

appuser 4216 6320 1936 O28:50 /usr/lib/ssh/sshd

root 4213 6320 3936 S28:59 /usr/lib/ssh/sshd

appuser 3886 1160 968 S13:09:24 /bin/sh ./startWLS.sh amdvt

[3556 byte] By [Vodola] at [2007-11-27 11:06:24]
# 1

> have box that has 2GB of RAM and 3GB of swap.

I assume you mean that you've configured 3GB of paging/swap space on disk.

> use ps command to check how much memory each program

> consumes. When I sum results I get numbers those are

> much smaller than memory and swap sizes: sum of the

> total size of the process in virtual memory - 866

> 664KB; sum of the resident set size of the process -

> 656 552KB.

If anything, those numbers are probably larger than the process size. Some pages in memory will be shared among them and will count in each process, even though it's only occupying one page.

> t the same time, when I'm checking free memory space

> by vmstat, I see only 35 840KB of free memory space

> out of 2 GB and 17 688RB of free swap space out of

> 3GB.

>

> When I try to run programs, in some cases, I get

> messages that there is not enough memory. Something

> like this:

> can't mmap anonymous memory: Resource temporarily

> unavailable

> not enough memory: Resource temporarily unavailable

> Not enough space

>

> How can I find out why I run out of swap and memory?

> Have anyone faced the same situations and what could

> be done to get back "lost" hundreds of megabytes

> without box rebooting?

> hat could be the reason of such situation?

#1 what is the output of 'swap -l' and 'swap -s'?

#2 Are any of your tmpfs spaces using large amounts of space?

(df -k -F tmpfs)

> [code]bash-2.05$ prtconf |head -2

> System Configuration: Sun Microsystems sun4u

Given the forum, can I assume this is Solaris 9?

> Memory size: 2048 Megabytes

> bash-2.05$ swap -l

> swapfile dev swaplo blocksfree

> /dev/md/dsk/d385,316 40994248464

> /swap/do_not_delete_swap_file ?,?16 2097136

> 166640

Okay, so the pressure is sufficient to cause the paging space to be heavily used.

> bash-2.05$ swap -s

> total: 4604496k bytes allocated + 99648k reserved =

> 4704144k used, 17768k available

The space is used, not just reserved.

So, since you see no sign of it in 'ps', lets look elsewhere. What are the results of:

df -k -F tmpfsand

echo "::memstat" | mdb -k

--

Darren

Darren_Dunhama at 2007-7-29 13:15:11 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 2

> So, since you see no sign of it in 'ps', lets look

> elsewhere. What are the results of:

>

> df -k -F tmpfsand

> echo "::memstat" | mdb -k

Unfortunately (for troubleshooting), the box has been rebooted (guys should work with it) and now the problem is not so obvious. So, I'll wait when the problem arises again (I believe, it will happen today or on Monday at the latest) and collect this information.

By the way, is there other command that could replace "echo "::memstat" | mdb -k". I don't have root access and the command returns output "mdb: failed to open /dev/mem: Permission denied". I'll try to be granted permissions but not sure if I get them.

> > have box that has 2GB of RAM and 3GB of swap.

>

> I assume you mean that you've configured 3GB of

> paging/swap space on disk.

yes, that's what I mean.

> Given the forum, can I assume this is Solaris 9?

yes.

bash-2.05$ uname -a

SunOS am-t-csmr1 5.9 Generic_118558-39 sun4u sparc SUNW,Ultra-80

Vodola at 2007-7-29 13:15:11 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...
# 3

Well, the tmpfs filesystem could have grabbed all the memory, but then again so could some other things.

No, I don't know any non-root equivalent for running the memstat stuff in mdb.

--

Darren

Darren_Dunhama at 2007-7-29 13:15:11 > top of Java-index,Solaris Operating System,Solaris Essentials - General Technical Questions...