NFS performance with Solaris 10
Hello,
We have been playing with one of the x4200s running s10u2, or snv_50 for that matter, and are getting terrible numbers from the NFS performance. Initially, we suspected it was just the ZFS filesystem on the back (which it was, though zil_disable made it a lot better), but even after exploring a little I am getting terrible numbers for NFS backed by UFS. Using afio to unafio a file on the disk gives:
Local:
afio: 432m+131k+843 bytes read in 263 seconds. The operation was successful.
Remote:
afio: 432m+131k+843 bytes read in 1670 seconds. The operation was
successful.
I have raised the ncsize to 1000000, and upped the server threads to 1024, but there does not seem to be an improvement.
The same thing on a linux box(ext3) turns in local times of 100 seconds and remote at 180 seconds. The differences in the local and remote numbers are just crazy. The difference in the ZFS is way worse:
Local zfs:
afio: 432m+131k+843 bytes read in 137 seconds. The operation was successful.
NFS -> ZFS:
afio: 432m+131k+843 bytes read in 2428 seconds. The operation was
successful.
I have started looking into dtrace for tracking the problem, but don't have much to report yet.
Using an older Solaris 9 machine with less memory (a v120) I get the following times:
Solaris 9 (v120):
afio: 432m+131k+843 bytes read in 652 seconds. The operation was successful.
compare this to a Solaris 10 machine (v240):
afio: 432m+131k+843 bytes read in 1802 seconds. The operation was successful.
In general read times from the drives over NFS are excellent, but the write performance over NFS is horrible compared to local write times. The problem is not the network though since other machines sharing the same network perform very well. And as noted Solaris 10 seems to do a lot worse than Solaris 9.
Any suggestions appreciated.
