SUN X6767A & X6768A HBA's throttled for Tape ACcess?
I have spent a lot of time benchmarking Tape SAN performance with an E15k
and several Crossroads 8000 FCAL routers front-ending an L700 tape library
with SDLT320 Drives and I am seeing something very strange yet 100% reproducible.
For those of you not familiar with the Crossroads 8000, it has 2 x 1Gbps FC ports, and
8 HVD SCSI.
If I use something like:
dd if=/dev/zero of=/dev/rmt/0cbn bs=512k
I will get about 37Mbps throughput to a single drive from a SUN X6767A or X6768A
HBA port. If I send more streams from that host bus adapter to the same FC port
on the Crossroads I end up with 2 x 23Mbps streams, or 3 x 13Mbps streams.
The interesting thing is that if I send those streams from the same HBA to a different
Crossroads FC Port, I will get 2x36Mbps (1 SUN HBA port to 2 Crossroads ports)
If I send from 2 Sun HBA ports to a single Crossroads FC Port I will get 2 x 36Mbps.
It seems as if the SUN HBA's have some kind of SCSI queue depth associated with
each SCS target that is probably set too low.
SunService says they don't deal with performance issues and referred us to SunPS.
Crossroads says their 8000 router works just fine and perhaps the Sun HBA is
trying to query the SDLT320 drives for speed on the SCSI bus and is throttling
througput to match up with this.
We don't think we see issues like this with JNI, but haven't had any cards available to test
with.
Question: Has anyone bumped into this 44Mbps per FCAL router port limitation with
SUN's HBAs, and/or can they send us any documentation that provides all the parameters
that can be set in the /kernel/drv/qlc.conf file? SunService claims to not have any documentation
on that config file, and we haven't seen anything in the Sun docs anywhere.
thanks,
Brian
[1911 byte] By [
bwolfe99] at [2007-11-25 23:12:29]
# 1
I will get about 37Mbps throughput to a single drive from a SUN X6767A or X6768A
HBA port. If I send more streams from that host bus adapter to the same FC port
on the Crossroads I end up with 2 x 23Mbps streams, or 3 x 13Mbps streams.
- So, your getting approx 40M/Sec for this one.
The interesting thing is that if I send those streams from the same HBA to a different
Crossroads FC Port, I will get 2x36Mbps (1 SUN HBA port to 2 Crossroads ports)
- Different Ports on the Crossroads (so two tape drives the other end of the bridge?) if so, approx 40M/Sec for this one too.
If I send from 2 Sun HBA ports to a single Crossroads FC Port I will get 2 x 36Mbps.
- This one sounds odd.
- Correct me if I'm wrong, cos I havnt seen SDLT320's for a while, but they are SCSI, and will swallow ~ 16M/Sec uncompressed and ~38M/Sec compressed (2:1 ratio), so what I think you may be seeing here is a bottle neck between the bridge and tape drive called "exceeding the ~40M/Sec of the BUS, or/and also how much data the tape drive can actually swallow that your forcing down its throat.
I would check the firmware on all of the devices too to ensure they are at acceptable levels.
I guess for your own piece of mind, is get the jni card on it and try. You should see the same sort of performance for each test your doing above.
deggs at 2007-7-5 18:02:17 >
# 2
Hello.I have the same problem too.What is solution?Thanks
# 3
different hardware setup, but similar (?) problem.
I have a 280R, solaris 8, up to date patches, with one channel
of a Sun dual 1Gb qlogic 2200 hba going to a Chapparal vfs226
fc-scsi router (test configuration). A tape library and a standalone
tape drive and scsi disks are on on its 6 lvd-160 scsi channels.
These are AIT-3 and Exabyte M2 drives, Qualstar library.
San and multi-pathing software/patches are installed.
If I attempt to saturate the channel with io, in a simple form,
dd if=/dev/zero bs=128k of=some-tape, I see 40MB/s to
a single drive. If I write to two drives, on separate scsi buses,
I see 2x22 MB/s. If I write to N drives, I see roughly 40/N
MB/s per drive. This is much lower than one would expect from a
1G Fibre channel, unless there were a -lot- of contention.
If however, I do even a bit of disk traffic to a scsi disk on
a different scsi bus on the fc router, I see 2x40 MB/s, up to
a total bandwidth of 96MB/s writing, or about 85MB/s reading.
(eg, 3x32MB/s, 4x24MB/s, and so forth). Originally noticed
using fsck, but tested with a C program with adjustable nanosleep();
the effect is immediate with even one raw disk read (of 512 bytes)
per second. The tape rates slow down within seconds of the disk
io traffic being stopped, and immediately increase when it is
started. iostat -x 5 is used to monitor the traffic.
Single stream results don't vary with disk traffic. Larger block sizes
(eg 256k) show faster rates (46MB/s), but the same effect.
The effect is seen with the application (samfs) with "real" data,
with about 30-50% non-compressible data rate improvement.
Library vendor claims this is a Solaris platform oddity,
and recommended scsi bus hardware analysis to debug it.
The observed disk effect makes me believe the claim.
# 4
I'm having the same problem, with a SUN HBA, Solaris 9, and a Qualstar library/fiber channel option. The best I've seen is 23 MB/s, which is then shared by up to 4 tape drives. Giving me 23/N MB/s. And this is all without using SAN. If I connect directly with scsi cables, I see upwards of 35 MB/s per drive.
Here's hoping someone is still reading this thread.
-Ryan
