Moving T3+ to FC-switch
Hi,
we have a 280R with a T3+ directly connected to via fiber (using an X6799A FC-HBA which is apparently a Qlogic 2200F). I need to migrate the content of the T3 to a newly installed EMC Clariiion CX300.
As a first step I tried to connect the server to the EMC FC-switch (DS-4400M) and connected the T3 to another port on the switch. The switch recognized both the server and the T3 (lights are green, config info shows up, altough the speed for the connection to the server was set to 4Gbps while I believe this card can only do 1Gbps so I changed that manualy). I've setup a zone on the switch so the server and T3 should be able to talk to eachother. However the server doesn't see the T3 anymore (format doesn't show the T3, '/etc/powermt display dev=all' says no devices found).
I assume the problem may be that the hba is controlled by the Sun QLC driver, while it should be (according to the EMC documentation) controlled by the Qlogic QLA driver. However when I install the qla driver it says there are no devices to attach to. The documentation I found from EMC and Qlogic describe what to do to make sure the QLA driver is used when installing a new hba but it doesn't say anything on what to do when the existing hba is already controlled by the QLC driver.
1. anybody any idea whether I should be able to see the T3 with the QLC driver (I'm not interested in talking to the EMC yet, just want to make sure I can reach my existing filesystem for now).
2. would removing the SUNWqlc... packages be enough to make the QLA driver see the hba? I'm a bit reluctant to remove those drivers as I don't know how to reinstall them in case everything fails. I realy need to be able to see the existing filesystem.
3. any other ideas on how to make the 280R see the T3 via the switch?
Thanks in advance,
Nico
[1874 byte] By [
Nico] at [2007-11-26 11:25:44]
# 1
I think you need to tell the T3B that it is not direct attached anymore and it is going via a switch.
I don't have a T3 in front of me to give you the details. Do a sys list and it should give you an idea of the setting you have. Sometimes mp_support (why, I have no idea but it can make a difference) and fc_topology need to be changed when introducing a switch. As I have never touched any hardware made by EMC (and hopefully never will) and there is no guidance on the Sun docs site about EMC switches, you will have to play with it.
I have had to make this change (with Qlogic and Cisco switches) when moving from direct attach to switches because the FC mode changes to something different. Sorry I am so vague but I have not done it in a while.
Also, sometimes the mode of the T3B can be a bit weird. You could change the switch port setting from being a F port to a FX port or perhaps a simple TL port.
HTH!!
Stephen
# 2
I think the server is more likely to be the problem. If I put the T3B in a zone with another existing server (X4200 running Solaris 10) the X4200 sees the T3B immediately (but 'format' claims I need to run fdisk first, I assume this is because the disk label on a Sparc based server is different from the disk labels used by x86 based servers).
On the switch I see the T3B port gets autoconfigured to be an F-port (Fabric port, configuration looks the same as for the other working devices connected to the switch) while the port to the server is an FL-port (Fabric loop). When I type 'probe-scsi-all' on the server in the boot monitor I get:
...
/pci@8,700000/SUNW,qlc@3
LiD HA LUNPort WWN-- Disk description --
Port Database command failed - unable to get WWN (4006) LiD = 0
...
Nico at 2007-7-7 3:41:11 >
# 3
I think you risk loosing data trying to move the T3+ array data between a SPARC system and a x64 system. I was looking at a project recently that was to migrate data from DAS to SAN, the best tool I could find was Sun StorageTek Data Replicator. The tools were bundled with the SAN arrays:
http://www.sun.com/storagetek/management_software/data_protection/data_replicat or/
# 4
I wasn't actually trying to move the T3+ from SPARC to x64, I just wanted to check whether another machine was able to see the T3+ via the SAN and the x64 is currently the only other server attached to the SAN.
However it's very interesting to know as the SPARC servers are planned to be upgraded to x64 next year. So I guess simply reattaching the assigned storage to the new servers will not be possible at that time.
Somehow the whole SAN thing looked a lot more promising when the sales guy proposed it :-/
Nico
Nico at 2007-7-7 3:41:11 >
# 5
Sorry, I was a little distracted when I made this post ( just returned from a 10km run ), I should have read your post more carefully. I will do some research on this for you this evening, but first can you give me some details on the OS release on the 280R servers ( I got the x64 and I guess it is Sol 10 ). It would be also helpful to know if you are using SFK ( SAN Foudation kit ). It should be possible to migrate to SAN from DAS, as long as there is no change in OS architecture ( eg SPARC to x64 ). I'm not familiar with the switch platform you are using ( EMC version of Sphereon 4400 Fabric Switch ), but could the server be seeing a WWN for the switch rather than the WWN in the device path?
Message was edited by:
m-lennon
# 6
First - It's widely known, I thought, that sparc and x64 architectures had different endian types. You can reuse the same storage but the data might not have the correct layout.
Second - When you move from DAS to SAN make sure you clean up the device links in the middle. You may be seeing some conflicts in the /dev and /devices tree.
Third - If an other host can see the storage but your target can not then you have a lun masking or zoning problem. Check the array and the switch. Also, make sure you have the latest SAN patches on the system.
Fourth - If you're moving to SAN move on Solaris 8 or 9 you need to run cfgadm -c configure <controllerport> to see the LUNs. Loop mode on those OS is automagic. SAN fabric needs to be explicitly configured.
# 7
As far as I am aware you cannot swap drives between Solaris SPARC and Solaris x86. There is a different partition type used in the Solaris x86 partition table. The only way to access the device with Solaris x86 would be to create a new partition table, if you remove the VTOC from the SPARC disk you will loose all data it contains. Before we can make further suggestions we need to get some more information, as I asked Nico to provide already.
# 8
You can swap the drives or luns as long as you relabel them, yes.
# 9
Yes, I agree, but only if there is no data on the drive, this is an unresolved issue with compatability between Solaris SPARC and Solaris x64. Anyway, I made a mistake earlier in the thread by assuming Nico was migrating from SPARC to x64, hence the confusion. Nico mentions just using the x64 system to test the T3Bs existance withing the SAN, I think Nico wants to move both 280R and T3B into the SAN. Solaris has issues with devices when the WWN has changed, there is a series of steps to reconfigure the system that I will provide if Nico confirms that this is the issue.
# 10
Hi guys,
thanks for all the replies I'll try to make things more clear:
1- I'm not trying to move from SPARC to x86. The x86 was just a quick test to see whether the issue was on the T3 side or the server side. As the x86 saw the T3 immediately I'm assuming the problem is on the server side
2- so what am I trying to do:
current situation:280R <- fiber -> T3B
new situation:280R <- fiber -> FC switch <- fiber -> T3B
I'm basically trying to migrate the content of the T3B to the new EMC SAN. I only have 1 HBA in the 280R so I figured if I attached the T3B and the server to the switch I could mount both the T3B and the EMC on the server and then start migrating the data from the T3 to the EMC gradually. Unfortunately I've been spending to much time in the network world where a 'switch' is mainly just a plug-and-play device :-). Note: the T3B installation was done by 'the-guy-before-me', this is my first experience with FC as you have probably already noticed.
3- what have I tried so far:
* hook up both the 280R and T3B to the switch, make a zone and hope it works... it obviously didn't
* put the T3B in a zone with an existing/working server (the x86). This worked immediately so I don't think I need to change anything on the T3B
* fool around with the port settings for the 280R on the FC switch, that doesn't seem to make any difference. According to the switch the port to the 280R is set to 'FL' (fiber loop) while all other ports which are in use are 'F' (fabric). when I reset the 280R to the ok prompt and try probe-scsi-all it doesn't see anything on the HBA.
4- what have I not tried so far: haven't tried changing the settings or driver on the 280R as this is an important production system and I need to be absolutely sure that if I can't get it running via the FC switch I can at least get it back to it's original state. I'm wondering whether removing the sun qlc packages and installing the qlogic qla driver will work/make any difference. However I don't know where the original qlc packages came from in order to be able to put them back in case nothing works anymore.
m-lennon:
- I actualy don't know whether I'm using the SAN Foudation kit, any idea how to check this? What's the package name?
- it don't think the server is seeing the WWN of the switch, it looks to me more like it's not seeing any WWN at all.
- the 280R is running Solaris 8
torreysun:
- I didn't know about the cfgadm -c configure <controllerport> command, I'll have a look at it
- 'SAN fabric needs to be explicitly configured': how do I do that (or is that what the cfgadm is about?)
Unfortunately my maintenance of last weekend expired, the server is back in production. I'm not sure when I will be able to do an new test yet.
Nico
Nico at 2007-7-7 3:41:11 >
# 11
Don't change any packages yet. Simple troubleshooting would suggest that the issue is with the 280R HBA or driver, but you took the system down to OBP and ran a probe, what happened? You mention that there was no communication with the switch or T3 " no WWN ". What is the HBA that is fitted to the x64 server ( 4Gb )? What driver are you using there ( qlogic or SUNW )? Also does the sphereon switch firmware support port configuration? Have you tried configuring the switch to throttle back to the 1Gb/s to match the configuration of the 280R HBA?
# 12
> Don't change any packages yet. Simple troubleshooting
> would suggest that the issue is with the 280R HBA or
> driver,
That's what I was thinking.
> but you took the system down to OBP and ran a
> probe, what happened? You mention that there was no
> communication with the switch or T3 " no WWN ".
It hangs for a while when checking the HBA and then gives
pci@8,700000/SUNW,qlc@3
LiD HA LUN Port WWN -- Disk description --
Port Database command failed - unable to get WWN (4006) LiD = 0
as I mentioned in one of my previous mails.
> What
> is the HBA that is fitted to the x64 server ( 4Gb )?
The x64 has a brand new QLA2340 (2Gb) while the one on the 280R is a very 1Gbps old card. I think the f-code is also very old (1.10). I tried installing some newer f-code from the Qlogic website but the tool failed.
> What driver are you using there ( qlogic or SUNW )?
Good point. I think I'm using the SUNW driver.
Nov 12 13:51:57 sun-x4200 qlc: [ID 308162 kern.info] Qlogic qlc FCA Driver v20060630-2.16 (0)
Nov 12 13:51:57 sun-x4200 pcplusmp: [ID 637496 kern.info] pcplusmp: pci1077,2312 (qlc) instance 0 vector 0x25 ioapic 0x5 intin 0x1 is bound to cpu 0
Nov 12 13:51:58 sun-x4200 qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(0): Link ONLINE
Nov 12 13:51:58 sun-x4200 qlc: [ID 694252 kern.info] NOTICE: qlc(0): Firmware version 3.3.117
Nov 12 13:51:58 sun-x4200 pci_pci: [ID 370704 kern.info] PCI-device: pci1077,100@1, qlc0
Nov 12 13:51:58 sun-x4200 genunix: [ID 936769 kern.info] qlc0 is /pci@1,0/pci1022,7450@2/pci1077,100@1
> Also does the sphereon switch firmware support port
> configuration? Have you tried configuring the switch
> to throttle back to the 1Gb/s to match the
> configuration of the 280R HBA?
Yes I tried forcing the port 1Gb/s and tried a few different settings for the port type. However it didn't make any difference.
Nico
Nico at 2007-7-7 3:41:11 >
# 13
See if you have the SUNWsan package installed. If you have that you probably have the SAN Foundation kit. If not...that explains why you can't see the fabric.
The FCode for the HBA would be available in a Sun patch if this is a Sun HBA.
The cfgadm command is documented in the FC docs, which came with teh SAN Foundation Kit, or the cfgadm_fp man page which on your system would be in /opt/SUNWsan/man (Thats from memory though so....don't quote me on that one.)
# 14
I'm not 100% sure that the issue is at the OS layer, drivers, SFK etc. won't affect communication at OBP. I suspect that there is something going on with the switch end as well. I'm going to do a little more research on this and post back later today.
# 15
I do have the SUNWsan package installed.
> I'm not 100% sure that the issue is at the OS layer,
> drivers, SFK etc. won't affect communication at OBP.
> I suspect that there is something going on with the
> switch end as well. I'm going to do a little more
> research on this and post back later today.
That's what I was thinking also. But maybe that's because of the old F-code?
Nico
Nicoa at 2007-7-21 15:32:20 >
# 16
I'm still searching for info on this, I won't be available until next Monday, but I will try to find out more over the weekend.
# 17
Sorry for taking so long to come back to this issue, I was very busy all week. Just looking back through the last few posts I notice that the FCode has been mentioned. I wonder if you misinterpreted what torreysun was suggesting about the firmware upgrade. I don't think it is possible to install the Qlogic firmware on the Sun HBA. In order to update the firmware you must install Sun patch 111853-04 ( installation instructions are included with the patch README ). I think that the older firmware only supports point to point and arbitrated loop communication, for switched fabric functionality, the HBA needs a firmware upgrade. I'm too busy to look through the obsolete patches for this card to understand if the older firmware you mention is limited to point to point and FC_AL, but I think it is a step in the right direction for this issue.
# 18
Hi m-lennon,
that patch is indeed not installed.
However, there has been a change of plans. Due to the problems getting the 280R to talk to the FC-switch and since a Solaris Sparc partition will not be usable on Solaris x86 and we were planning to migrate the whole system to Solaris x86 next year anyway, we have decided to stop trying to hook up the 280R to the EMC but migrate to a new x86 system directly. Otherwise we will have to move the content of the raid twice anyway (it took me almost a day to copy the raid to another system last time).
So this means I will not try to reconfigure the system anymore (altough I do want to reuse the HBAs in some other systems, but that will be done after the move).
As you guys put a lot of effort in solving my problem however the case never got solved completely I decided to award half of the Duke dollars anyway if that's ok with you: 4 for m-lennon and 1 for torreysun (not as active as m-lennon but I did get some very useful feedback out of it)
Thanks a lot for the help!
Nico
Nicoa at 2007-7-21 15:32:20 >
