Metadb on a SAN
Using Solaris 8, I have 2 JNI card connections, using SDD software, to an IBM Shark SAN.
Currently, I'm using Solstice Disksuite with metadbs on the internal disks.
The SAN is working fine.
I have set up an extra metadb replica pair in a slice on the far end of one of the virtual disks (vpath1h).
Again, everything shows up fine until I reboot.
At this point, my metadb -i shows:
Mp unknown unknown /dev/dsk/vpath1h
Mp unknown unknown /dev/dsk/vpath1h
The only thing I've spotted is an early message in /var/adm/messages:
unix: [ID 935016 kern.warning] WARNING: Failed to process interrupt for vpathdd0 due to down-rev nexus driver pseudo0
Have I failed to initialise the SDD software before solstice kicks in to read the metadb replicas?
How can I fix this?
[842 byte] By [
axassl] at [2007-11-25 23:19:31]
# 1
when you run format, what does it list regarding the characteristics of disk vpath11?
if you haven't set the disk parameters, then you can't use the disk on the sun system... (this is my guess based on the info received :)
if that doesn't help, what kind of san is it?
fred
fim32 at 2007-7-5 18:07:39 >
# 2
Hi Fred,
Pastes from format:
This is the cxtcdx for the device
6. c2t0d0 <IBM-2105800-.358 cyl 4170 alt 2 hd 64 sec 256>
/pci@9,600000/JNI,FCR@1/sd@0,0
and the second path to the same disk.....
25. c3t0d0 <IBM-2105800-.358 cyl 4170 alt 2 hd 64 sec 256>
/pci@9,600000/JNI,FCR@2/sd@0,0
And this is the vpath entry for the pair
44. vpath1a <IBM-2105800-.358 cyl 4170 alt 2 hd 64 sec 256>
/pseudo/vpathdd@0:1
Which disturbingly shows vpath1a rather than what I would have (innocently) thought should have been vpath1c (all 19 vpaths show as vpathxa).
The partition table for the disk shows as follows
Current partition table (original):
Total disk cylinders available: 4170 + 2 (reserved cylinders)
PartTagFlagCylindersSizeBlocks
0usrwm0 - 3833.00GB(384/0/0)6291456
1usrwm384 - 416829.57GB(3785/0/0) 62013440
2backupwu0 - 416932.58GB(4170/0/0) 68321280
3 unassignedwm00 (0/0/0)0
4 unassignedwm00 (0/0/0)0
5 unassignedwm00 (0/0/0)0
6 unassignedwm00 (0/0/0)0
7usrwm4169 - 41698.00MB(1/0/0)16384
I've never seen the SAN. I just know it as an "IBM Shark" San. The device is definitely an IBM ESS 800 - model something or other - the guy who knows is off today so doesn't have access to his documentation.
Be aware that everything else about the San is operating totally normally, loading several databases at the moment. The metadbs on it have remained OK overnight. I have another Solaris system accessing another set of disks on it, getting the same problems
Ian W-J
# 3
Hi Fred,
Pastes from format:
This is the cxtcdx for the device
6. c2t0d0 <IBM-2105800-.358 cyl 4170 alt 2 hd 64 sec 256>
/pci@9,600000/JNI,FCR@1/sd@0,0
and the second path to the same disk.....
25. c3t0d0 <IBM-2105800-.358 cyl 4170 alt 2 hd 64 sec 256>
/pci@9,600000/JNI,FCR@2/sd@0,0
And this is the vpath entry for the pair
44. vpath1a <IBM-2105800-.358 cyl 4170 alt 2 hd 64 sec 256>
/pseudo/vpathdd@0:1
Which disturbingly shows vpath1a rather than what I would have (innocently) thought should have been vpath1c (all 19 vpaths show as vpathxa).
The partition table for the disk shows as follows
Current partition table (original):
Total disk cylinders available: 4170 + 2 (reserved cylinders)
PartTagFlagCylindersSizeBlocks
0usrwm0 - 3833.00GB(384/0/0)6291456
1usrwm384 - 416829.57GB(3785/0/0) 62013440
2backupwu0 - 416932.58GB(4170/0/0) 68321280
3 unassignedwm00 (0/0/0)0
4 unassignedwm00 (0/0/0)0
5 unassignedwm00 (0/0/0)0
6 unassignedwm00 (0/0/0)0
7usrwm4169 - 41698.00MB(1/0/0)16384
I've never seen the SAN. I just know it as an "IBM Shark" San. The device is definitely an IBM ESS 800 - model something or other - the guy who knows is off today so doesn't have access to his documentation.
Be aware that everything else about the San is operating totally normally, loading several databases at the moment. The metadbs on it have remained OK overnight. I have another Solaris system accessing another set of disks on it, getting the same problems
Ian W-J
# 4
Just curious, are you using mpxio?
scavy at 2007-7-5 18:07:39 >
# 5
MPXIO - Yes. We originally took off mpdi and mpdix, intending to use Veritas, then decided (possibly only for the time being) to use Disksuite as our requirements were simple (almost a "give me one huge disk"), the San is raiding our disks inside itself already, and our current skill/experience was disksuite.
So we reinstalled mpdi and mpdix off the cd.
As you can see, we're on a San learning curve at the moment.
Am I starting up my Disksuite before SDD has initialised the vpaths? Would that explain it?
Ian W-J
# 6
so vpath1h, where you are trying to put your metadb is not a physical device? rather, vpath1h does not represent a drive, but is an allocation of data from a larger pool of drives?
i'm not sure that you can put your metadb on a virtual device like this. if your virtual device was a 1-1 mapping onto an actual drive, i could see this working, but the metadb doesn't fit into a regular formatting?
strange, i'm assuming for vpath1h, h refers to the 7th slice? that slice starts and ends on cylinder 4169; well the block size is apparently big enough...
ok, what does your mddb.cf file look like? it should be in /etc/lvm
fred
fim32 at 2007-7-5 18:07:39 >
# 7
Oh yes, IBM Total Storage Enterprise Storage Server "Subsystem Device Driver User's Guide" strongly suggests I CAN do this (SC26-7478-01).
It devotes a whole chapter to using SDD on a Sun system and specifically cites Solstice Disksuite and setting up metadbs.
vpath1a refers to disk instance one, slice 0; vpath1c is slice 2, and h becomes slice 7.
Slice 7 is 8 megs, each metadb replica is only just over 500k.
From my Solaris machine's point of view, it's supposed to be seeing a regular bank of disks out there - it's not supposed to realise that they're only virtual devices built on a huge data area.
mddb.cf:-
#metadevice database location file do not hand edit
#driver minor_t daddr_t checksum
ssd716-311
ssd71050-1345
ssd1516-319
ssd151050-1353
vpathdd 1516-736
vpathdd 151050-1770
I already had a perfectly usable set of metadb replicas on my machine. It just seemed wise to add another pair of replicas on the San - another path of availablity to a metadb replica, plus a capability for another system (say a DR machine) to pick up the data on the same San and use it.
Ian
# 8
sorry, that was just a shot in the dark.
y'know, i just re-read your original post, and i think you have something there. look in the /etc/system file and see what MDD has put in there. i'll bet you can forceload the sdd driver, so that the system has access to the san disks prior to md startup.
worth a shot?
fred
fim32 at 2007-7-5 18:07:39 >
# 9
Fred,
You're a hero! That looks like it.
In /etc/system I see a set of forceloads for:
misc/md_*various (which is disksuite)
drv/*various
then finally (and separaely from the rest)
drv/vpathdd(which are my multi-paths)
I'm guessing that then vpathdd should be before the misc/md stuff
It got automatically placed there by the install of SDD software (IBMdpo)
As the machine's in use, I need to allocate a slot to try it out.
Watch this space......
Many thanks again!
Ian
