3510FC performance monitoring at individual spindle level
Hi!
I've got a particular LD on my 3510 which is running very hot. The host shows 100% utilizaiton of the disk in iostat and the server's IOWAIT is in the 70-90% range.
My config is one 3510 single controller, direct attached to 2 hosts. 1 v480 and 1 intel box with Redhat Linux. Each host is presented with with 2LDs from the 3510. Both servers are Oracle DB servers and one LD is the data file storage and one LD is the archive log store.
My question is, is there a way to see what the usage of the individual spindles in the LD is like? I am curious to know if there is a layout problem where one disk is getting beat up. Or possibly the controller is saturated?
Thanks!
-Bryan
[719 byte] By [
bmcgowan32] at [2007-11-26 6:58:25]
# 1
%wio in and of its self can be very misleading. If you have multiple processors each considers its self to be waiting for all outstanding I/O so the metric is inflated on MP systems, and if there is nothing else to do you get %wio even if the I/O sub system is working beautifully, so it is essentially an indication of idle time.
Also with a HW RAID LUN 100% busy is also misleading, and not the same as 100% utilization. By analogy you can drive a Formula-1 race car at 1 mph and keep it 100% busy but it is far from 100% utilization. So it is with HW RAID LUNS. A single thread of 64 KB sequential write can keep the target 100% busy, but if it抯 a 4+4 RAID 1+0 device, each spindle is only ~ 25% busy. With 4 threads in this example, and say a 64 KB stripe unit, you can keep 4 spindles ~ 100% busy, but they are still not 100% utilized. A single spindle will saturate somewhere > 1 and < 16 or so threads, depending on the I/O size and access type. However for good response times you want to operate the spindle in a comfortable range not at 100% utilization.
The key to understanding the load on the internal spindles of a HW RAID LUN is noting the response time you are seeing at the host LUN level as it reflects the response time of the individual spindles. Response time varies directly as load level so measuring or calculating the load level at the LUN and dividing by the number of data drives in the HW RAID LUN is of interest to gauge how hard you are hitting the drives. The true load level is ~ [ (response time * IOPS) / %b]. So if you have 0.020 response time and are seeing 200 IOPS at 25% busy, the true load level is ~ (0.020 * 200) / 0.25 = 16. iostat actv would report something around 4 as it is off by %b as far as the true load level, i.e. the average load level not counting idle time. If you are 100% busy then the load level is as report by iostat actv, which will be ~ (response time * IOPS) where IOPS is rds + wrs. Once you know the load level divide by the number of spindles in the RAID, like 4 in a 4+4, and this is ~ the load level per spindle. By the way the relationship between load level, response time and completions a second is called Littles Law.
Another thing to look for is %w (not %wio) this tells how much time is spent on the secondary wait queue which is only used when the controller queue is full. The Solaris default is 64, so if you see any %w at all you know you have load levels peaking over 64. Again if it抯 a 4+4 RAID, 64 / 4 = 16 which will saturate just about any spindle with just about any I/O size access type combination.
There are some very powerful tools available to model I/O workloads and decompose them into I/O size access type combinations with respect to load level that will answer these questions precisely and empirically at the host level. Please feel free to contact me for more information.
HTH,
Dave Fisk, ORtera Inc.
