Quorum Disk in a Dual Site Cluster
We have in general a cluster setup with two data centres, in each of the two data centres we have one of the cluster nodes and a SAN. Both nodes are connected to both SANs over two fabrics, so each node sees all disks. The disks are then mirrored with SLVM, and one of the SAN disks server as quorum disk, just a simple plain vanilla setup.
But now we get a problem if the data centre on whose SAN the quorum disk resides goes down completely: the cluster will fail as the remaining node doesn抰 not get enough votes. Is there another way than having the quorum disk in a third location to survive such a failure?
Someone claimed the Veritas Cluster would be able to handle such a situation.
Tks
Fritz
[732 byte] By [
Tom_Tiger] at [2007-11-26 9:17:43]
# 1
Hi,
It all boils down to the question whether there is an algorithm that can distinguish a node outage from a communication outage (split brain). I think there is not.
If you look at the analogy with 3 managers located in 2 sites, how would they decide whether there is communication problem or a people problem?
There might be products that claim they can. If you want to be on the safe side, you need a majority quorum algorithm, as it is implemented in SC3.x. And if you only have 2 sites, there is 50% chance, that you lose your majority.
Now there are 2 alternatives: either you can live with this situation and the manual recovery procedure, or you have to find a 3rd room for the 3rd vote. Or you use an additional firecell using different power grids or UPSs, or whatever to make the qourom survive.
With the next cluster release there should be an alternative for the in-cluster quorum, but still it would need a 3rd site!
Regards
Hartmut
# 2
Well, I know, you can't solve the split brain problem with just 2 locations.
I have now talked with our Veritas experts, they explained me how Veritas Cluster handles this:
- If a node detects a failure on both heartbeats at the same time, it assumes the other node has gone and takes over all the services.
- If both of this connections go down at different times, a node assumes a problem with the network and does not take over services.
In order to have this reliable working you have to make certain that:
- both heartbeats us separate switches, pwer sources and connections
- the cluster node is connected to the same power source as the switch so it goes down if both switches loose their power
This method is not absolutely bullet proof as it may result in a split brain situation if both heartbeat connections are interrupted for some reason. But in reality chances of this happening are much lower than a data enter outage, for example due to an power outage and a not working UPS or whatever.
Has anyone an idea how we could get such a behaviour with a Sun cluster?
# 3
I like the wording "it assumes ..". Would you bet your business on assumptions that something worked?
OK, here is the deal. If chances that a DC outage occurs are very, very low, why not rely on manual intervention in that rare case to clean up the configuration instead of betting on a solution which is known not to be bullet proof?
Hartmut
# 4
Hi Hartmut,
I completely agree to your statement from a technical point of view.
So in the end it is the choice between higher availability or lower risk of data corruption.
We hade a huge power outage in one of our DCs some time ago, so this is the main disaster scenario in the heads of the management. Thus they favour the higher availability and consequently the more expensive Veritas Cluster.
Fritz
# 5
Why not considering adding a UPS to the storage device that holds the QD? That is probably much cheaper than a VCS license!
With the UPS the QD would survive and you just needed a process to add another QD on the surviving DC and to remove the old one.
Hartmut
PS: There should be some interesting functionality coming with the next SC release, btw. that could help in this case, if you do not like the UPS idea.
PPS: I doubt that with the VCS approach you have higher availability. I'll ask my availability numbers specialist to come up with some numbers.
# 6
Hi Hartmut,Regarding to your PS, what kind of new cluster features will this be, and when can we expect it.Regarding to your PPS, thanks in advanceFritz
# 7
Would a quorum (device) which is not part of the cluster help. This would mean that no strict cluster connectivity and other requirements had to be met.
This might be available in the near future. Sorry, to be so vague. But predictions about software availability have to be imprecise.
H.
# 8
Yes, i guess a quorum device would be the solution of this problem. I have been informed about the Sun Cluster 3.2 Beta, but i have currently no resources to test it, but it is promising.Fritz
# 9
Hello,
The main problem is to make the difference between a "room down" and a "split brain". With only two rooms, in case of "split brain", the node witch owns the quorum disk will take over, the other one will just panic.
But, in case of "room down", if the room hosts the node that owns the quorum disk, the second node will panic. Thus the whole cluster will be down.
As far as I know, the only reliable solution to this is to have a third small room, just to put a small machine, like a Fire v210. This machine will act as a third quorum node but will never run any application, so does not need to be connected to the SAN's.
eric
