swapped out LWPs waiting for processing resources
Hello,
I have a 5.10 Generic_118855-19 i86pc i386 i86pc server. For the last couple days vmstat have been reporting 43 swapped out LWPS.
root@server # vmstat 5
kthrmemorypagedisk faultscpu
r b wswap free re mf pi po fr de sr m0 m1 m1 m1insycs us sy id
0 0 8 25266920 6540720 238 1217 603 4 4 0 13 2 1 2 0 12945 5471 4850 3 4 93
0 0 43 71081680 7031384 119 962 520 0 0 0 0 0 0 0 0 9597 2465 5291 1 3 96
0 0 43 71070596 7019932 91 567 500 0 0 0 0 0 0 0 0 5430 1890 2552 1 2 97
0 0 43 71065348 7014764 79 1033 523 0 0 0 0 0 0 0 0 5011 2173 2064 1 2 97
0 0 43 71067672 7017480 62 328 467 0 0 0 0 0 0 0 0 4234 1595 1848 0 2 98
0 0 43 71066604 7016632 75 345 430 0 0 0 0 0 0 0 0 3910 1177 1651 0 1 98
The number of LWPs stays the same. It doesn't decrease. This started when one of the apps (p4d) allocated 40GB of memory. That process was killed.
Do you know how can I find out/trace who's threads are still swapped out?
Many thanks and best regards,
Piotr
# 1
I've spoken to a couple of engineers about this over time. None of them have had a great solution.
Presumably we should be able to use 'pmap -x' to find processes that have no active private mappings. Those would be the ones that are swapped out. But in practice I've found it difficult to reconcile the numbers.
Good luck.
--
Darren
# 2
Hi ,
Even I have the same doubt.
pmap could give only the shared memory consumed by the process.
but I also want the CPU usage per LWPs.
I have another doubt as follows,
kthrmemorypagedisk faultscpu
r b wswap free re mf pi po fr de sr m0 m1 m5 m7insycs us sy id
0 0 81 7153248 876096 0 0 0 182 182 0 0 0 0 0 1 627 4971 703 4 3 93
0 0 81 7153248 876072 1 0 0 192 190 0 0 0 0 0 0 627 5037 694 5 3 93
0 0 81 7153248 876040 34 34 0 184 184 0 0 0 0 0 0 660 5971 755 7 3 90
here we see the pagged out (po) processes is same as the freed process , but it never showed those many pagged in LWPs.( pi ).
Thanks,
San.
MIDLa at 2007-7-12 9:09:03 >
# 3
> Hi ,
> Even I have the same doubt.
> pmap could give only the shared memory consumed by
> the process.
> but I also want the CPU usage per LWPs.
I don't understand the question. 'pmap' shows memory usage. If you're interested in CPU usage then you'd use 'prstat' or 'ps'. Both can show LWP usage.
A swapped out process will be swapped out entirely, so no LWP will have any usage.
> I have another doubt as follows,
> kthrmemorypagedisk
> faultscpu
> swap free re mf pi po fr de sr m0 m1 m5 m7in
>sycs us sy id
> 0 81 7153248 876096 0 0 0 182 182 0 0 0 0 0 1
>627 4971 703 4 3 93
> 0 81 7153248 876072 1 0 0 192 190 0 0 0 0 0 0
>627 5037 694 5 3 93
> 0 81 7153248 876040 34 34 0 184 184 0 0 0 0 0 0
>660 5971 755 7 3 90
> here we see the pagged out (po) processes is same as
> the freed process , but it never showed those many
> pagged in LWPs.( pi ).
Now I really don't understand what you're asking. The figures under pi and po are KB of memory, not LWP counts (the vmstat man page should show that).My assumption is that this memory was paged in in the past, and you're just seeing some freeing in your example.
--
Darren
# 4
Hi Darren,
Thanks a lot for looking into this. I fixed that problem by reboot. Unfortunately, we had the same issue again (killed 2 processes - p4d perforce). My question is: Is it safe to leave the server in such state:
kthr
r b w
0 0 47
0 0 47
0 0 47
and wait till next reboot or we are looking at potential performance issues?
I will play with pmap -x
Best regards,
Piotr
# 5
No, in general there's no performance impact after the memory issue is addressed.
Any swapped out processes remain swapped out until they need to run, so your 'w' column will probably not drop back down to zero.
If you could identify the processes, then you could either kill/restart them, or perhaps "poke" them in some manner that would make it swap back in.
--
Darren
# 6
I also just tried something. I was wondering if you could delete the swap arena to force the system to unswap the processes.
Didn't work. In fact I'm getting a failure from 'swap -d' that says not enough space, when I'm quite certain there's enough. I suspect that the swapped out blocks may be locking down the swap arena.
# prtconf | grep Mem
Memory size: 128 Megabytes
(128MB RAM)
# swap -s
total: 52624k bytes allocated + 6768k reserved = 59392k used, 2093992k available
(Only 59MB in use, so no space needs...)
# swap -d /dev/dsk/c0t0d0s1
/dev/dsk/c0t0d0s1: Not enough space
But I can't release the space...
--
Darren
