Suggestions for a redundant Oracle setup
Hi,
I don't know if this is the right place to ask this, but I am asked to research different methods of configuring a system for running Oracle RAC.
We have previously used two Sun v20z servers and a shared SE 3310 for this. But now we want to use two storage arrays, for complete redundancy.
We are thinking two SE 3510 for this. But we have no experience with FC and are fishing for suggestions how to accomplish this.
For example, how do we mirror the two arrays, is it possible to use the built in RAID controller somehow (one or two in the arrays) ? Or should we use something like SVM/Cluster suite ? Veritas ? Or can Oracle 10g ASM do what we need ?
Also, do we need FC switches int this setup, or are there enough ports on the 3510 to support this ?
How about Multipathing ?
Appreciate suggestions for such a solution or pointers to example setups, both soft and hardware suggestions.
Tor
[967 byte] By [
tarein1] at [2007-11-25 23:02:52]
# 1
Here is a link to the SE 35xx array best practice guide:
<a href="http://docs.sun.com/app/docs/doc/816-7325-19" target="_blank">http://docs.sun.com/app/docs/doc/816-7325-19</a>
If you can tell me what parts you want to use can provide a little more configuration information. Most of the Sun Cluster configuration information is available through the documentation site too, there is limited information available on Oracle products external to Sun. The best place for that is <a href="http://otn.oracle.com/" target="_blank">http://otn.oracle.com/</a>
BTW what operating system are you planning to use, Solaris 10 x64?
# 2
Hei,
Thanks for the reply.
I have looked through the Best practices guide for the 3510, and the only thing I found related to my setup was this (well, as far as I understood it) :
"The entry-level configuration for a Sun StorEdge 3510 FC array or Sun StorEdge 3511 SATA array uses only one RAID controller. If this configuration is used, two singlecontroller arrays should use host-based mirroring to ensure high reliability, availability, and serviceability."
Does this mean that with two dual controller arrays mirroring can be performed by the arrays themselves ? And the OS sees the two arrays as one device ?
Regarding parts we want to use, we have not decided anything really except for 2*3510 and most likely 2*v40z for servers with whatever HBA cards that is recommended (2 or 4 HBA's pr. server ? ).
We normally use RedHat Enterprise Linux 4, but for this we are considering Solaris x64 instead.
Tor
# 3
I will update the thread with more detailed configuration rules for the SE 3510 tomorrow, it's a bit late to get into it now ( keep in mind that the JBODs are out of the question if you are running Sun Cluster 3.x ). You'd be looking at a data/mirror set up between the two arrays, are you familiar with traditional Sun HA systems ( T3 or photon )? I am gald to hear you are looking at a V40z, the V20z has limited I/O subsystem and you could run into issues if you were to use >=2 HBAs for better redundancy. I have a configuration guide for the V40z and 3510 that I made for a customer, so I can give you details on recommended HBAs etc., but you will also have guidance from your Sun pre sale technician when you go to make the purchase.
# 4
Ok, lets get into this a bit. You must provide me with some more information, if you are using RHEL then there is no need to discuss the operating systems points in this thread. Also Sun Cluster 3.x is not supported with RHEL and these guidelines are specific to Sun Solaris x64 and Sun Cluster 3.x ( that's not to say that the hardware configuration will not work with another operating system! ).
True HA using, say a single 3510 will require two HBAs in each node in the cluster. Two options are available the SG-XPCI1FC-EM2 and SG-XPCI1FC-QLC. The latter is the card selected for use with Solaris 10 3/05/RHEL 3.0 - 4 on V40z when attached to Sun switches and 3510 arrays. There are some restrictions when using the Emulex card on the V40z so we won't discuss that here. Install the adapters in any slot from 4 through 7 ( eg slot 5 and 7 ). Next parts are the switches, for a small configuration the recommended options are 2 * SG-XSW8-2GB ( two eight port switches ). Keep in mind though that if you are using two arrays with dual controllers and you want better scalability you should choose 2 * SG-XSW16-2GB or bigger. Sun also sells Brocade and McData switches, if you are interested in these switches please post back and I'll make a suggestion.
Call the servers Server 1 and Server 2. Call the switches Switch 1 and Switch 2. Call the HBAs port A and B. Server 1 port A is attched to Switch 1 and port B is attached to Switch 2. Server 2 port A is attached to Switch 1 and port B is attached to Swich 2.
You can see this information clearly outlined in chapter 5 of the bbm:
<a href="http://docs.sun.com/source/816-7325-19/ch5_san_co nfigs.html" target="_blank">http://docs.sun.com/source/816-7325-19/ch5_san_configs.html& lt;/a>
There is also the DAS option, which can be configured without switches. The ports are configured in a similar way to the SAN type, but there is a specific patern when attaching to the array controllers. Depending on the growth of your database, this method may prove to be less scalable than the SAN configuration. DAS with two controllers would be limited to 8 expansion units, if you are using RAID 10 this will limit you to 54 drives ( under 8TB using 146GB drives ). See chapter 4 of the bbm.
<a href="http://docs.sun.com/source/816-7325-19/ch4_das_co nfigs.html" target="_blank">http://docs.sun.com/source/816-7325-19/ch4_das_configs.html& lt;/a>
Multi pathing software will be manditory in this configuration with Solaris x64.
When it comes to two arrays, I am not 100%, I use this setup myself, but the arrays are mirrored using SVM. Two arrays with a single controller can be used, but you must mirror the arrays, the actual configuration depends on a number of factors, I'll do some more research and I'll update again later.
<b>UPDATE</b>
As mentioned before, you will have to clarify what OS and Cluster software you want to use before we proceed. I will only provide information on Sun Cluster on this forum. The other products mentioned ( Red Hat Cluster and Oracle 10g ASM ) are not covered on the Sun Storage hardware support forum unless there was a hardware related issue with these products. But I would encourage anyone considering implementing Oracle RAC on Sun hardware to use the entire Sun package StorEdge>Sun Fire>Solaris>Sun Cluster. I'm sure Stuart or Brian could give some input here as they would have experience supporting hardware for Oracle in the financial services industry.
# 5
Hi again,
Thanks for the tips so far.
Regarding switches, we do not need any future expansion of the system, so I assume we will not need switches then ?, and go for the DAS option.
Will will use Solaris 10 x86, so in this case multipathing software means SUN Traffic Manager ?
When it comes to Cluster software, I think we will prefer to use Oracle ASM for the shared storage, and let Sun do mirroring of the arrays, either in hardware or software. Is this possible ? But, I am open to suggestions.
I think we are supposed to setup the system in a Sun lab together with Sun engeneers later, so maybe going Sun all the way is better ?
Tor
# 6
The decision not to use switches should be carefully considered, in fact your data growth should be a major factor! Many early database solutions were deployed on DAS, but many companies experience a steady growth in storage, needs for storage consolidation and migration to SAN is inevitable. Moving data off a DAS into a SAN can be difficult and will require disruption of service at some point, how much will in cost to take the database off line for 4 hours ( this figure would be reasonable if nothing goes wrong! imagine the loss if that time increased to 8 - 12 hours! )?
Last night I looked at some information on Oracle ASM, from a DBAs perspective it is a very good solution. Being able to manage storage resources with the features available in ASM has to be a plus for any DBA. It also seems to offer great improvment in performance. Are you a DBA ( ocp )? To me ASM seems to be a way of managing your storage resources in a single node environment and not a cluster. You will need to implement a multi pathing driver and Solaris 10 x64 has SAN foundation software included in the release. You can find information in the Solaris 10 documentation.
<a href="http://docs.sun.com/app/docs/doc/819-0139" target="_blank">http://docs.sun.com/app/docs/doc/819-0139</a>
I can't see that ASM can provide the same features as Sun Cluster, even though the V40z can be configured up to 8 way, Sun Cluster will allow you to scale your resources well beyond two initial nodes ( up to 800 CPUs! ). Possible configuration scenario would have to be well thought out, something that I was thinking about would be to create a number of hardware RAID 1 LUNs on the disk arrays, make these resources global with Sun Cluster, then manage the LUNs as devices with ASM ( stripe LUNs ). It's hard to say how this would work out, it's a bit 'pie in the sky' and it could turn out to be a nightmare to manage... array management, Sun Cluster, Solaris and Oracle ( RAC and ASM ) - could over complicate the solution. I thought of this as a way to try and ustilize the T3 platform, which only supports RAID 0, 1 and 5 your 3510s have a wider range of supported RAID.
When you are setting the testbed solution with Sun you will come up with a solution that works for you, solutions on the drawing board don't always work out smooth in the field. The plus for the all Sun + SAN solution is that you will have a very scalable solution, so down the road if the solution needs extra resources you can basically add nodes and more disks, this could prove much cheaper than upgrading the hardware to bigger systems. The downside is that if you are a DBA this will mean that you will either have to train up on all these extra system features or employ an extra systems administrator to look after the Sun side of things, DBAs and sys admins always seem to have a conflict of interests and neither is ever willing to help the other out!
# 7
I don't know what you've decided, and don't care what Sun tells you _DO NOT USE SINGLE CONTROLLER CONFIGS_
It is a very stupid design to even allow it. I could go through personal details etc..
1 example is that you can not do online firmware updates. Its just a bad way to do it.
# 8
Thanks for the input Travis, Sun Cluster supports two array config with a single controller on each array. In this environment updates can be carried out on an array that has been temporarily taken out of service, this is a RAS feature that Sun Cluster provides. Also firmware update on disks will require the system to be taken out of service completely. BTW from Sun sales perspective a single controller set up is usually not offered, this is something that would be requested by the customer and this is the reason for SC supporting this minimum configuration, so this is not something that Sun will tell you. Please tell us about your experiences.
# 9
Hello Travis,very interesting point !Btw. is Troy (who registered as tdknabe) a relative or is your family name just as common as mine ?Michael
maal at 2007-7-5 17:54:50 >
# 10
maal can we keep on the subject of computer storage please?
# 11
Hi,
It is still not clear to me how the two arrays could/should be mirrored. Since it was recommended not to use the single controller option here, can mirroring be done in hardware without the involment of the OS when using dual controllers ?
Or else I cannot see that the dual controller option gives me that much more (well, except maybe expandabillity)
Tor
# 12
If you are using two arrays with a single controller you must use host based mirroring to provide redundancy. One tool for the job is Solaris Volume Manager which is part of Solaris 10 x64.Veritas can also be used.
# 13
Ok, So, if I understand correctly, if I use two dual controller arrays, I can use hardware mirroring ? And I dont need to use SVM ?Which is better ? And how will the two arrays show up on the hosts, as one device ? (for each logical disk)Tor
# 14
As far as I can tell SE 3510 does not have a feature that you can mirror a pair of arrays, even with dual controllers. There are interfaces on the controller, but these are for expansion units. Using a volume manager is a good setup, though not something you seem to be keen on. The other choice you have is to go for a midrange storage system.
# 15
OK, thanks for clearifying that. SVM it is then...
I don't have much experience with SVM other than simple software mirroring with 2 system disks, so I kind of hoped it was possible to do this in hardware.
Thank you for having patience with me, asking all these questions, it's getting clearer now. But I have some new ones:
Well, lets asume I have two single controller arrays, and use SVM to mirror them. Then, will I cable one cable from each array to each machine (just asuming DAS now) ? Then, would I need to use Multipathing software ? Wouldn't SVM take care of that ?
Is it correct that the machines then will see each array as a logical drive (for simplicity here), and the application (Oracle) will see the final mirrored device (for example /dev/md/dsk/d10) ?
What other software is needed in addition to SVM to enable cluster functionallity (since we are talking about shared storage here) ? Is this included in SVM or Sun Cluster software ?
And last, I don't have access to FC arrays here now, only SE 3310's can I theoretically build a test system with two 3310's and two servers with 2 SCSI cards each to get familiar with the software setup ?
Tor
# 16
The cables will need to be attached to one port on each controller, the operating system will need a path to both arrays. So using the example from earlier, you'll need Server 1 port A attached to Array 1 and Server 1 port B attached to Array 2 ( in the best practice guide figure 4-5 shows the configuration for multipathing on a dual contoller array, your configuration will be similar to this, but double check the attachment sequence for the controllers ). Multpathing software will be needed and is manditory for 3510.
You will see a device in Solaris that will be the LUN that has been created using the 3510 controller. With volume manager you will use metainit command to create a logical volume out of the LUN, you can then use metattach to mirror each volume ( or array ). At that point Solaris will indeed show a volume ( something like /dev/md/dsk/d<i>n</i> ), so say it's mounted on /u01, that's all you'll see when you run the Oracle installation program ( I can't remember Oracle installtion stuff, I haven't been involved with any Oracle databases since 2003 with Oracle 8/9i ).
Sun Cluster will be needed, it doesn't only provide storage availability, but also the ability for provide failover for the node hardware as well ( SVM cannot ). So lets say your cluster is running away, but your primary node goes down from a bad CPU, Sun Cluster will rotate the servers and you application will stay 5-nine. Also SC can be used to scale nodes.
Yes you can use the 3310 to emulate the theory of the 3510 setup, you will need a path to both arrays from each host though!
This will require dual HBAs.
Here is the 3310 best pratice guide:
<a href="http://docs.sun.com/source/816-7293-13/bestpracti ces.html" target="_blank">http://docs.sun.com/source/816-7293-13/bestpractices.html< ;/a>
There are a few clustering rules though, are your 3310s JBOD or RAID?
# 17
Thanks,So, I should use 4 HBA's for each server for multipathing with dual controller arrays ?My 3310 is RAID, and yes I meant using two SCSI adapters. But I can't test multipathing with this configuration, am I right ?Tor
# 18
For configuring two V40z in a cluster testbed on the SE 3310 RAID, I would use two X4422A cards in each server. This card will give you two SCSI interfaces ( per HBA ) on each server and four additional network interfaces that can be used for the cluster interconnect and better network capacity ( 4Gb/s trunk to the switch to application clients ).
I am not 100% on the multipathing, I don't have experience with this type of storage ( mainly SAN ), but I don't think it's needed. The actual bus configuration for the "dual array configuration" shows channel 0 on each nodes' two HBAs attached to a port on the 3310 Host/Drive I/O module. So there is no attachment to the array Controller Module itself.
Also keep in mind that when you are building a HA cluster using the 3310 as shared storage, that the 3310 only supports a max of 2 nodes per cluster.
