Solaris 10 (amd64) and spontaneously high load average problem
I have a "server" running Solaris 10. It's an AMD64 server with 1 GB of RAM and a pair of 160 GB SATA disks on an Asus NForce4 board. The system uses SVM to mirror root, and ZFS for home directories, which are shared out via NFS and Samba to a closed network of 5 machines. That's pretty much all the machine does.
The problem is that I recently ran "smpatch update" on the machine to update to a more recent version of code than I was running (security patches, etc were needed, and this way seemed easy to do). After rebooting, the system is fine for up to 24 hours, sometimes only 12. After that, the load average climbs to the 4.0 range and sits there. vmstat shows the system with no running I/O and the CPU ~95% busy. prstat shows no single task using anything more than 0.1% of the CPU.
I have next to no experience with combing the kernel for information on what it's doing "under the hood", but I did see that when the system was "bad", I have 13000-ish "nthreads" in two places (under cpu). When the system is freshly booted, that number is closer to 1800, but seems to slowly climb over time. For instance, 30 minutes after booting back up and letting the system sit there doing nothing at all, it's up to 2100. I don't know if that's normal for a thread count in the kernel to do, but with no userspace jobs using the CPU, that's the only place I can look.
Any suggestions as to how to go about diagnosing and fixing this ?
