3510 FC transfer speed
I have Sun StorEdge 3510 with 1 RAID Controller and 5x 72GB 10000rpm HDD.
I'm trying to check the performance of this array.
I connect only one HDD without any RAID Levels.
Write Policy: Write-Back
Cache Optimization: Sequential I/O
I'm copying 6GB file from servers internal SCSI HDD to my FC HDD.
The speed is ~15MB/s.
When copying from my FC HDD to the same FC he peed is a same.
Is it correct?
I was expecting a much more speed.
May be I have a wrong settings?
[540 byte] By [
Macho] at [2007-11-25 23:03:20]
# 1
this might help..
<a href="http://www.storageperformance.org/results/b00005_Sun_SPC2_executive-summa ry_r1.pdf" target="_blank"> http://www.storageperformance.org/results/b00005_Sun_SPC2_ex ecutive-summary_r1.pdf</a>
this was found on the 3510 web page.
haroldkarl
# 2
What server are you attached to? ( HBA etc. )
# 3
haroldkarl thanks but speeds in document are much more than my speed.It's just testing server Sun Enterprise 250.After testing I'll connect this array to SUnFire 280R.HBA is Emulex PCI-X 2Gb Single Channel FC.
Macho at 2007-7-5 17:55:13 >
# 4
The internal SCSI bus the E250 is about 40MB/s, that's your bottle neck there.
# 5
But when I'm copying file from from 3510 to 3510 a speed is the same.
Macho at 2007-7-5 17:55:13 >
# 6
Given the size of the file you are transfering your throughput figures will depend on the filesystem configuration as well. What OE are you running?
# 7
Solaris 8
Macho at 2007-7-5 17:55:13 >
# 8
Have you created a standard UFS on each LUN?
# 9
Yes.I have created a standard UFC.newfs device-name (without parameters)
Macho at 2007-7-5 17:55:13 >
# 10
I'll follow up with more information on this tomorrow, I don't have enough time tonight.
# 11
OK Thank you.I'm waiting for your replay.
Macho at 2007-7-5 17:55:13 >
# 12
There are a few factors that need to be considered with the tests that you are currently performing. Firstly using cp or mkfile is not going to test your storage properly, granted these commands may perform better on the servers internal storage, but the kernel will allocate system resources differently for devices attached to the system. Secondly using a single disk from the array is not going to test the actual performance of the array, disk arrays are designed to spread the data over many spindles. Thirdly the size of the file you are testing compared to the performance limitations of the server hardware, the E250 has a slow PCI bus, slow SCSI bus, slow UPA bus and slow memory. This server was designed to attach to UltraSCSI and FC100 arrays. If I were to design a system running Solaris 8 that was going to be shifting data at >=6GB/file I would be looking at a V890 or E2900 with +16GB of memory to perform anyway reasonably.
Can we take a few steps back and look at what application is going to use the array. How are you going to configure the array ( RAID level? ). What configuration of your server. From there you'll need to create an I/O profile for the application and then tune the filesystem to suit the application. From that point we can start to test the array throughput.
# 13
m-lennon,
At first I want to thank you for your relays.
I'll try to explain you a situation.
It's my first contact with FC HDD-s.
I had heard a lot of about speed of FC HDDs.
So I have expected a HUGE speeds.
But I have recieved only 15MB/s.
We purchase this storage to connect it to SunFire280R server.
But at first I must test a performance of this sotage on our test server (as you know E250).
I can't connect this storage to the working system without a prove that the speed will be high.
After some of my tests (I show you results) I think that problem is not in storage or in HDDs and I should seach for it in the server itself.
But I don't know what can be the reason of these speed.
There is an old PCI Bus 66Hz, but it's throughput is 532MB/s.
HBA is new: Emulex PCI-X 2Gb Single Channel FC.
Internal SCSI ..., but I think I must consider it's speed only when I'm reading or writing from internam SCSI drives.
What esle?
Ok Now results of my tests:
The highest speed for reading from storage HDD I get by this command:
dd if=filename of=/dev/null bs=512000
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t40d1
166.8 0.0 78575.1 0.0 0.0 0.8 0.0 4.9 0 82 c2t40d0
The highest speed for writing to storage HDD I get by this command:
dd if=/dev/zero of=filename bs=1024000 count=.....
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t40d1
0.0 126.4 0.0 57154.7 0.0 0.7 0.0 5.6 0 67 c2t40d0
Macho at 2007-7-5 17:55:13 >
# 14
Again I'll have to update you tomorrow, I had to test a few CPUs and a pile of memory tonight and I just did't have time to look at this. Right off I can see that there may be issues with how you are testing the storage throughput. At a glance, correct me if I'm wrong, you are taking a 6GB file and breaking it down into small peices and transfering it to a device. Again I think dd is not the tool for the job here, and nowonder you are getting similar throughput figures!
# 15
The best way to check the speed of the array is with the raw device. If you are looking for maximum sequential bandwidth you will want to use large I/O, this is especially true with a single threaded test like dd.
Try dd if=/dev/null bs=1024k of=/dev/rdsk/cxtxdxsx count=6144
Better yet, get vxbench, vdbench, or some other load generator so you can use multiple threads. Try two or three threads of 1 MB I/O to a RAID-0 with 5 disks with a 512 KB stripe unit. This will get very close to the maximum bandwidth of the array when you have enough I/O Bus to handle it.
You will need to set maxphys in the kernel to 1048576 and use newfs C 8 to get 1 MB I/O from the UFS filesystem.
HTH,
Dave
# 16
Thanks for the input Dave, I am not 100% on the Veritas benchmark tool vxbench, but the Sun vdbench is only available internal to Sun. I would say that you will have to pay Sun services test your storage with SWAT and vdbench.
There are a few opensource I/O benchmark tools also. We recently started working with some mixed UNIX systems - Solaris, HP-UX and Linux, so I needed a filesystem benchmark tool that could be used on various platforms. Here is a link:
<a href="http://www.iozone.org/" target="_blank">http://www.iozone.org/</a>
I am in the very early stages of using this tool, so I can't say how useful it actually is.
Out of various opensource tools I looked at, iozone has a wide range of features and presented decent documentation.
I'd be interested to hear how the raw device test went though, Macho please post the results.
# 17
Ok Thank you for your replays.I'll test this tool tomorow and I'll post you results.There is one thing more I want to know.Can someone explain me, why these 3 copying commands show me different results: dd, cp, mc(midnight commander).
Macho at 2007-7-21 14:28:42 >
# 18
You are welcome.
The cp(1) command uses mmap(2) for read, and write(2) for write, with a 256 KB I/O.
The dd(1) command uses the input option ibs=n or bs=n for the read I/O size, and obs=n or bs=n for the write I/O size.
Both are single threaded.
I/O performance is a function of load level and response time for a given workload composition. By workload composition I mean the relative portion of each I/O Size, Access Type combination. For example 20% 8 KB Random Read (RR) + 80% 32 KB Random Write (RW) is an example of a workload composition.
Given a workload composition there is a fairly linear relationship between response time and load level. It is the weighted sum of the response times for the components of the workload composition at the given load level. For example, if the load level (thread count) is 5, and the response time for 8 KB RR at a load level of 5 is 15 ms. and the response time for 32 KB RW at a load level of 5 is 30 ms., than a good estimate of the response time for 20% 8 KB RR + 80% 32 KB RW at a load level of 5 is:
0.20 * 0.015 + 0.80 * 0.030 = 0.027 = 27 ms.
Throughput is then a function of load level and response time by Littles Law. IOPS = Load_Lavel / Response_time.
So in the above example the capability at 100% busy is:
5 / 0.027 = 185 IOPS.
This is composed of 20% 8 KB and 80 % 32 KB so the bandwidth is:
0.20 * 185 * 8192 + 0.80 * 185 * 32768 = 5,152,768 = 4.9 MB/sec
(1 KB = 1024 bytes)
I dont know if iozone supports raw devices, I believe it is a filesystem benchmark only (but I am not sure).
You can get vxbench from Veritas:
<a href="ftp://ftp.veritas.com/pub/support/vxbench.tar.Z" target="_blank">ftp://ftp.veritas.com/pub/support/vxbench.tar.Z</a>
# 19
There are the results of vxbench:
Benchmark= buffered_reads
#thds #apps blocksizeiocount filesizethruputCPU sec
111 1 1323 KB/s0.00
112 1 2729 KB/s0.00
114 1 41506 KB/s0.00
118 1 82647 KB/s0.00
118 2 165105 KB/s0.00
118 3 245870 KB/s0.00
118 4 327714 KB/s0.00
118 5 407208 KB/s0.00
118 6 488586 KB/s0.00
118 7 568832 KB/s0.00
118 8 649720 KB/s0.00
118 9 7210869 KB/s0.01
118 10 8011997 KB/s0.00
118 11 8811958 KB/s0.00
118 12 9613499 KB/s0.00
118 1310413120 KB/s0.01
118 1411213518 KB/s0.01
118 1512014274 KB/s0.00
118 1612814641 KB/s0.00
118 2016017938 KB/s0.01
118 2419221313 KB/s0.01
118 2822423560 KB/s0.01
118 3225625645 KB/s0.00
118 4032030565 KB/s0.00
118 4838417932 KB/s0.01
118 5644820291 KB/s0.01
118 6451222609 KB/s0.01
118128102435042 KB/s0.02
118256204850508 KB/s0.03
118512409662271 KB/s0.05
1181024819269541 KB/s0.10
11820481638474650 KB/s0.20
11840963276879659 KB/s0.39
11881926553668995 KB/s0.88
1181638413107268790 KB/s1.70
1183276826214448119 KB/s3.39
1186538452307255644 KB/s6.87
118131072104857668835 KB/s14.06
118262143209714464815 KB/s28.60
Benchmark= buffered_writes
#thds #apps blocksizeiocount filesizethruputCPU sec
111 1 1456 KB/s0.00
112 1 2809 KB/s0.00
114 1 41788 KB/s0.00
118 1 82700 KB/s0.00
118 2 165013 KB/s0.00
118 3 247199 KB/s0.00
118 4 328864 KB/s0.00
118 5 4011594 KB/s0.00
118 6 4813809 KB/s0.00
118 7 5614230 KB/s0.00
118 8 6417282 KB/s0.00
118 9 7219016 KB/s0.00
118 10 8020548 KB/s0.00
118 11 8822204 KB/s0.00
118 12 9622789 KB/s0.00
118 1310420328 KB/s0.01
118 1411221005 KB/s0.00
118 1512022078 KB/s0.00
118 1612822912 KB/s0.00
118 2016027935 KB/s0.00
118 2419230387 KB/s0.00
118 2822434527 KB/s0.01
118 3225637260 KB/s0.00
118 4032040105 KB/s0.01
118 4838443440 KB/s0.01
118 5644846371 KB/s0.01
118 6451247559 KB/s0.01
118128102461177 KB/s0.02
118256204841486 KB/s0.03
118512409643206 KB/s0.05
1181024819244133 KB/s0.10
11820481638443931 KB/s0.20
11840963276844122 KB/s0.38
11881926553642270 KB/s0.83
1181638413107241740 KB/s1.71
1183276826214439306 KB/s3.91
1186538452307238568 KB/s7.93
118131072104857638296 KB/s16.24
118262143209714437495 KB/s33.23
Benchmark= overwrites
#thds #apps blocksizeiocount filesizethruputCPU sec
111 1 1335 KB/s0.00
112 1 2765 KB/s0.00
114 1 41378 KB/s0.00
118 1 83396 KB/s0.00
118 2 166465 KB/s0.00
118 4 3212520 KB/s0.00
118 8 6419698 KB/s0.00
118 1612826538 KB/s0.00
118 3225639247 KB/s0.01
118 6451253349 KB/s0.01
118128102466359 KB/s0.01
118256204844712 KB/s0.02
118512409645464 KB/s0.05
1181024819244919 KB/s0.10
11820481638445012 KB/s0.19
11840963276844452 KB/s0.37
11881926553644287 KB/s0.79
Benchmark= fsync
#thds #apps blocksizeiocount filesizethruputCPU sec
111 1 1297 KB/s0.00
112 1 2585 KB/s0.00
114 1 41079 KB/s0.00
118 1 82105 KB/s0.00
118 2 163797 KB/s0.00
118 3 245521 KB/s0.00
118 4 326894 KB/s0.00
118 5 407724 KB/s0.00
118 6 4810114 KB/s0.01
118 7 5610599 KB/s0.00
118 8 6411155 KB/s0.00
118 9 7212107 KB/s0.00
118 10 8011837 KB/s0.00
118 11 8812518 KB/s0.00
118 12 9612642 KB/s0.00
118 1310410988 KB/s0.00
118 1411211714 KB/s0.01
118 1512012075 KB/s0.00
118 1612811827 KB/s0.01
118 2016014324 KB/s0.00
118 2419216647 KB/s0.00
118 2822418913 KB/s0.00
118 3225619232 KB/s0.00
118 4032020773 KB/s0.01
118 4838421992 KB/s0.01
118 5644824063 KB/s0.01
118 6451225417 KB/s0.01
118128102436001 KB/s0.01
118256204839277 KB/s0.03
118512409640229 KB/s0.05
1181024819240969 KB/s0.11
11820481638441853 KB/s0.20
11840963276841897 KB/s0.43
11881926553642215 KB/s0.85
1181638413107240173 KB/s1.82
1183276826214438402 KB/s4.03
1186538452307237788 KB/s8.16
118131072104857637491 KB/s16.75
118262143209714435999 KB/s34.45
Benchmark= sync_writes
#thds #apps blocksizeiocount filesizethruputCPU sec
111 1 1270 KB/s0.00
112 1 2601 KB/s0.00
114 1 41129 KB/s0.00
118 1 81828 KB/s0.00
118 2 163022 KB/s0.00
118 4 323843 KB/s0.00
118 8 644689 KB/s0.00
118 161284639 KB/s0.01
118 322564395 KB/s0.01
118 645124378 KB/s0.02
11812810244260 KB/s0.05
11825620484247 KB/s0.11
11851240967117 KB/s0.22
118102481924205 KB/s0.43
1182048163844224 KB/s0.94
Benchmark= mmap_reads
#thds #apps blocksizeiocount filesizethruputCPU sec
11 256 125632293 KB/s0.00
11 256 251242715 KB/s0.01
11 256 376849581 KB/s0.00
11 256 4102451877 KB/s0.01
11 256 5128051770 KB/s0.01
11 256 6153654684 KB/s0.01
11 256 7179255826 KB/s0.01
11 256 8204855990 KB/s0.01
11 256 16409659777 KB/s0.02
11 256 32819260626 KB/s0.04
11 256 641638461569 KB/s0.09
11 2561283276861762 KB/s0.18
11 2562566553661525 KB/s0.36
Benchmark= mmap_writes
#thds #apps blocksizeiocount filesizethruputCPU sec
11 256 125634525 KB/s0.00
11 256 251244173 KB/s0.00
11 256 376848314 KB/s0.01
11 256 4102451676 KB/s0.01
11 256 5128053513 KB/s0.00
11 256 6153651897 KB/s0.02
11 256 7179254901 KB/s0.02
11 256 8204855235 KB/s0.00
11 256 16409658276 KB/s0.03
11 256 32819260112 KB/s0.04
11 256 641638460722 KB/s0.10
11 2561283276846330 KB/s0.20
11 2562566553657477 KB/s0.38
Benchmark= cached_writes
#thds #apps blocksizeiocount filesizethruputCPU sec
118 1 893630 KB/s5.47
118 2 1695960 KB/s10.47
118 4 32100674 KB/s20.49
118 8 6486085 KB/s1.46
118 1612880821 KB/s3.17
118 3225686613 KB/s5.96
118 6451252610 KB/s11.99
118128102452681 KB/s20.37
118256204851502 KB/s40.03
Benchmark= cached_reads
#thds #apps blocksizeiocount filesizethruputCPU sec
118 1 8164771 KB/s3.19
118 2 16157947 KB/s6.64
118 4 32150593 KB/s13.93
118 8 64158942 KB/s26.38
118 16128160231 KB/s52.35
118 32256152697 KB/s109.25
118 64512156157 KB/s214.88
1181281024155255 KB/s6.76
1182562048154538 KB/s13.58
Macho at 2007-7-21 14:28:42 >
# 20
Thanks for the data.
What is the LUN in this case?
It would be interesting to see a RAID-0, 5 disks with a 512 KB stripe unit, with vxbench nthreads=3, iosize=1024k,...
You might also try nthreads=10, iosize=512k
A very nice feature of vxbench is it provides multi threaded I/O however, you have only run single threaded tests so far...
# 21
No update David, did you call Sun? One thing I'd like to add for the benefit of any future readers - testing of your storage performance should involve tests that replicate the application I/O profile that will be using the storage. These tests should be carried out along with the raw device performance tests. A storage system that has a great raw throughput could perform poorly with a badly configured filesystem.
