quorum device question
I have a 2 node test cluster up and running.
It has hitachi open-9 LUN attached to both nodes, via fibre channel, as its quorum device.
I am unclear about a few things with regards to this.
Can both nodes write to this device simultaneously? Probably not.
I have it mounted on both nodes, and tested writing to it... just a simple mkdir issued from each node in turn.The respective nodes can only see their respective writes.. and one node has a disk error show in its syslog, requesting I run fsck.
This was not mounted with the -g option.
I would like to understand how I can make this quorum disk into something useful where the shared configuration files can reside, with any writes being immediately visible to both nodes.
[770 byte] By [
diggles] at [2007-11-26 11:42:07]
# 1
Hi,
you misunderstood the purpose of the quorum device. It is a vote needed in case of a cluster reconfiguration. It can be accessed by all cluster nodes attached to it, but in a way that is not controlled by the user. So, what you are trying does not make sense.
On the other hand, the device, thatholds the quorum, can be used for a filesystem. You can mkfs it and mount it and use it. No problem.
If you want to access the same filesystem from more than one node you need a shared or global or cluster filesystem. There is a special mount option "global" that allows you to do it. Then you can write from any node to it, and you'll see any writes immediately on all nodes.
What you are trying is subject to fail miserably, as the normal filesystem code does not handle global filesystems.
I recommend reading the Suc Cluster Concepts Guide on http://docs.sun.com
Regards
Hartmut
# 2
Hartmut,
Sometimes it is neccessary to attempt apparently illogical things in order to understand better how they work. When it is not just pure lazyness, there actually can be value gained from asking the stupid questions. Unfortunately not everyone in IT have the personality trait that will allow themselves to "look stupid".
All too often, the obvious gets overlooked. The simplicity of things can get drowned in technical terms, so that "concept guides" are so boring to read that people can't help skipping over essential information.
I had read the concepts guide - I knew what a quorum device is for, and about global filesystems. The quorum disk does not always successfuly mount globally from the vfstab on bootup, so I was trying other things out.
By the way,
If I init 5 on one node, the following can happen on the other node:
df: cannot statvfs /global/.devices/node@2: I/O error
(this happened to node 1 when I did init 5 on node 2)
Of course, it does not happen if I manually unmount it, and mount the global devices directory for the node that is staying up.
Apparently Sun Cluster does not manage this automatically?
One thing is for certain. It does not allow me to have both dirs:
/global/.devices/node@1
/global/.devices/node@2
simultaneously mounted.
Any chance someone could lower their intellects enough to explain this one to me? :-)
Take it easy,
Diggles the Solaris Jester.
# 3
Hi,
I am really sorry for being too harsh. I was confused by your terminology.
And I fully agree that looking beyond the typical use cases and asking the "stupid" questions often reveals useful insight. So let's continue.
I was mainly confused by "mounting the qourum device". The qourum device is configured on a shared LUN. This same LUN can be used for a shared filesystems. If that does not mount correctly, there is something wrong.
Looking at my test cluster it does mount both /global/.devices/node@1 and ...@2 in paralle. And actually it has to.
/global/.devices/node@1 on /dev/did/dsk/d1s7 read/write/setuid/devices/intr/largefiles/logging/noquota/global/xattr/nodfrati me/onerror=panic/dev=3bc0027 on Mon Nov 6 14:34:56 2006
/global/.devices/node@2 on /dev/did/dsk/d7s7 read/write/setuid/devices/intr/largefiles/logging/noquota/global/xattr/nodfrati me/onerror=panic/dev=3bc00e7 on Mon Nov 6 14:52:26 2006
So, the question is what is wrong with your cluster? If you are using SVM did you make sure that both initial /globaldevices metasets had different ids? Can you post the output of scdidadm -L? Maybe we can find the problem here.
Regards
Hartmut
# 4
Hartmut,
Thanks and no problems. Here is the output.
# scdidadm -L
1node1:/dev/rdsk/c0t0d0/dev/did/rdsk/d1
2node1:/dev/rdsk/c1t0d0/dev/did/rdsk/d2
3node1:/dev/rdsk/c1t1d0/dev/did/rdsk/d3
4node1:/dev/rdsk/c1t2d0/dev/did/rdsk/d4
5node1:/dev/rdsk/c1t3d0/dev/did/rdsk/d5
6node1:/dev/rdsk/c2t50060E80035B0001d0 /dev/did/rdsk/d6
6node2:/dev/rdsk/c2t50060E80035B0011d0 /dev/did/rdsk/d6
7node2:/dev/rdsk/c0t0d0/dev/did/rdsk/d7
8node2:/dev/rdsk/c1t0d0/dev/did/rdsk/d8
9node2:/dev/rdsk/c1t1d0/dev/did/rdsk/d9
10node2:/dev/rdsk/c1t2d0/dev/did/rdsk/d10
11node2:/dev/rdsk/c1t3d0/dev/did/rdsk/d11
From node1's vfstab:
/dev/md/dsk/d30 /dev/md/rdsk/d30/global/.devices/node@1 ufs2noglobal
From node2's vfstab:
/dev/md/dsk/d30 /dev/md/rdsk/d30/global/.devices/node@2 ufs2noglobal
These entries were autogenerated by the Sun Cluster installation process.
# 5
Hi again,
there is an often overlooked sentence in the installation guide at
http://docs.sun.com/app/docs/doc/819-0420/6n2rlnnb5?a=view
"If the device is a local device to be used to mount a global-devices file system, /global/.devices/node@nodeid, the metadevice or volume name for the mirror must be unique throughout the cluster."
In your config you use a metaset named d30 on both nodes. Now if that gets imported on both nodes, you have a name conflict, that produces the problems and error messages that you see. You should change the name on one node and your problems will go away - hopefully.
BTW: The same problem exists with VxVM. There you have to reminor these global devices after the setup, so that there are no conflicts.
In general: all objects that might be used globally must have unique names.
Hth
Hartmut
# 6
HI,
Well, take a look of the following :
6 node1:/dev/rdsk/c2t50060E80035B0001d0 /dev/did/rdsk/d6
6 node2:/dev/rdsk/c2t50060E80035B0011d0 /dev/did/rdsk/d6
Have the same /dev/did/rdsk/d6 so it can be used as quorom device because the 2 nodes can see the same LUN, and additionally it can be used as global or cluster filesystem.
From node1's vfstab:
/dev/md/dsk/d30 /dev/md/rdsk/d30 /global/.devices/node@1 ufs 2 no global
From node2's vfstab:
/dev/md/dsk/d30 /dev/md/rdsk/d30 /global/.devices/node@2 ufs 2 no global
Look and see the d30 is the same in the 2 nodes use the metarename the d30 to another device and you will see the 2 gobal devices mounted in the 2 nodes. You will need to see this 2 global devices mounted when the cluster is running in a normal state.
FOr Hamsert, dude if you don't want help, don't help but you do not need to offend anybody, if you think the question is stupid for your guru knowledge just skip it and go for other one. That is my personal opinion
JoshD
JoshD at 2007-7-7 11:48:42 >
# 7
Thanks Hartmut and Josh,
Renamed it with metarename and now both nodes are mounting their global device directories in the cluster.
However, I can't get the quorum disk to mount using vfstab properly on either node.
After reboot:
df: cannot statvfs /global/u1: I/O error
/dev/md/dsk/d30486M3.6M434M1%/global/.devices/node@1
/dev/md/dsk/d70486M3.5M434M1%/global/.devices/node@2
Both nodes have this in the vfstab:
/dev/did/dsk/d6s2/dev/did/rdsk/d6s2/global/u1ufs2yeslogging,global
I am sometimes able to mount it manually though:
umount /global/u1
mount -g /dev/did/dsk/d6s2 /global/u1
Although, it often returns:
umount: /global/u1 busy
Especially when this intermittant error occurs during bootup on the node being rebooted:
Could not stat: ../../devices/pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@1/fp@0,0/ssd@w50060e80035b00 11,0:c,raw path not loaded.
No such file or directory
/dev/md/dsk/d30486M3.6M434M1%/global/.devices/node@1
/dev/md/dsk/d70486M3.5M434M1%/global/.devices/node@2
/dev/did/dsk/d6s26.8G13M6.7G1%/global/u1
My dilemma is that I would like to store common service configurations on the quorum disk, but these services are failing to start on reboots when the quorum disk is not properly mounting, because they cannot access their configs.
# 8
That sounds strange. You might have some inconsistencies in your did strcuture. You can try to run:
scdidadm -C
scdidadm -r
scgdevs
This should remove entries for non-existing devices and recreate the global name space.
It looks a bit strange, that you use SVM metasets for the other filesystems, but a raw device for u1. Can there be another inconsistency here?
On the other hand, you write, that a manual fglobal mount succeeds from time to time. What do you see if it fails? I/O error?
Do you see any interesting error messages in /var/adm/messages?
Could you post the output of scdidadm -L and scstat -D?
Thanks
# 9
Hi Guys,Sorry about the delay in my response here.I've caved in and logged a support call with Sun and they're helping me out with getting this working. Will let you know what the fix is when we find it :-)
