defunct process

> Hi all

>

> I found a lot of defunct process in one of our

> solaris server. It is occupying around 95% of the

> memory.

What is 'it'? Defunct processes are dead and don't occupy any memory.

What leads you to believe that the defunct processes are responsible?

> I killed all those defunct processes, but

> memory is not yet released.

You can't kill defunct processes, they're already dead. All you can do is convince their parent to reap their exit status, or kill the parent so that 'init' does the job.

> Anybody have an idea about

> how to release that memory without rebooting the

> system

What method are you using to deermine memory in use? Is this a Solaris 9 machine? Are you using 'ps' and looking at the virtual memory size?

--

Darren

Darren_Dunham at 2007-7-6 18:59:31 >

> I am able to kill those defunct process

You have a very different Solaris system than I do then. Are you sure they're not just getting reaped between when you see them and when you look after the kill?Can you show a 'ps' output for them?

# ps -ef | grep defu

root 3199 3198 00:00 <defunct>

# kill -9 3199

# ps -ef | grep defu

root 3199 3198 00:00 <defunct>

> and i am able to see the memory using prstat -a -s

> size

We can look at it a litte more directly with ps. Can you try this on one of your defunct processes and show the output?

# ps -p 3199 -o pid,s,vsz,rss,args

PID S VSZ RSS COMMAND

3199 Z00

Notice that on this machine a defunct process occupies no virtual memory and also no RAM pages.

--

Darren

Darren_Dunham at 2007-7-6 18:59:31 >

Yup. I believe the "small amount of system memory" that it mentions is in fact the allocation for the exit status.

You will almost certainly run out of PIDs before you notice the memory consumed by the zombies.

Heck, I've got a box here I can abuse, let's try it. It's even *painfully* low on memory, being an Ultra-10 with 192MB and Solaris 10 and I'm logged in with a JDS desktop. "max_nprocs" on this machine is 2922.

I ran this command:

# perl -e 'while (1) { $a = fork(); if (!defined $a) {print "Unable to fork. $! Sleeping.\n" ; sleep 30; exit; } elsif ($a == 0) { $b = 7 ; $b++; exit; } }'

(The math with $b is just there so that the child does *something* besides exit, hopefully forcing at least one page allocation that should be returned).

When it finished, 'prstat' went from 75 processes to 2919 (pretty close to 2922).

vmstat went crazy during the run (and load was over 50 temporarily), but it settled down.

0 0 0 413928 343600 52 0 0 0 0 0 0 0 0 0 406 979 402 6 4 90

[job started here...]

42 0 0 397208 29632 155 3871 117 0 0 0 0 15 0 0 0 440 1058 363 13 46 41

134 0 0 368048 30256 253 6847 37 335 447 0 237 7 0 0 0 415 1540 324 18 82 0

[ran a while... and settled down during the 30 second sleep]

0 0 0 413472 357600 52 0 0 0 0 0 0 0 0 0 401 964 249 5 3 92

0 0 0 413472 357600 52 0 0 0 0 0 0 0 0 0 402 984 266 4 3 92

[parent exits and all the zombies are reaped]

0 0 0 413800 367360 53 0 0 0 0 0 0 0 0 0 402 3267 250 6 16 78

0 0 0 413888 369840 52 0 0 0 0 0 1 0 0 0 406 901 258 4 3 93

So, yes. A bit of memory was returned. For over 2800 zombies, something on the order of 400 kb was returned on exit. That's about 140 bytes per process.

'vmstat' isn't necessarily completely accurate and something else on the machine may have affected the output. Assume that the above could be off by 100% or more and it's still not a signifcant amount..

--

Darren

Darren_Dunham at 2007-7-6 18:59:31 >