growfs not expanding filesystem
Encountered an interesting problem that I think is worth sharing. Had a client who requested
increasing their RAID1+0 filesystem. The environment is as follows: adding 4x36GB disks, Solaris 9,
Solaris Volume Manager (aka DiskSuite). Once we had the OS see the new drives, started the process
to grow the filesystem. In Solaris Volume Manager, it's actually pretty straight forward. In fact,
it's really only two steps:
1) add 2 of the disk partition to each of the submirrors (e.g. metattach d41 c2t0d0s2 c2t1d0s2; metattach d42 c2t2d0s2 c2t3d0s2)
2) grow the filesystem (e.g. growfs -M /mount_point /dev/md/rdsk/d40)
The problem we had was due to one of the original drives wasn't seated correctly and did not come
up on boot up and resulted in the metadevice mirror we were trying to grow to go into "Needs maintenance" state.
Once we enabled the metadevice, of course it had to sync up. Since we were on a tight schedule, we decided
to proceed with the configuration of growing the filesystem. When a resync is in progress, the mirror cannot
be expanded until the resync completes. You'll get a message similar to this:
metattach: sd146234: d40: growing of metadevice delayed
Of course, the sub-mirrors will grow without any problems. We weren't paying too much attention and thought
that we were successful in growing the mirror and proceeded with the growfs portion. And lo' and behold, the
filesystem still had the original size. Being pressed for time and not understanding why growfs wasn't working
we decided to reboot the server in hopes that it would fix the problem. Again, we still weren't paying attention
to the resync that was still running in the background. When the server came back up, the resync started again
and we tried to see whether we could run another growfs to expand the filesystem. Same results. At this point,
we knew there was no way we were going to meet our maintenance window and started looking at the problem a little
more carefully. After 20 minutes, we noticed that the resync finally completed. As we looked at metastat, we finally
realized that the mirrored metadevice still had the original size whereas the submirrors expanded correctly.
The metastat out with the sizes looks something like this:
#metastat d40
d40: Mirror
Submirror 0: d41
State: Okay
Submirror 1: d42
State: Okay
...
Size: 355618566 blocks (169 GB)
d41: Submirror of d40
State: Okay
Size: 497867148 blocks (237 GB)
...
d42: Submirror of d40
State: Okay
Size: 497867148 blocks (237 GB)
...
We suspect that the reason the submirrors expanded correctly but not the mirror was due to the resync that
was in progress and badly timed reboot. The steps to correct the problem are as follows:
1) detach the second submirror
# metadetach d40 d42
2) unmount the filesystem
# unmount /raid
3) delete the mirror
# metaclear d40
4) re-create the mirror and attach submirror
# metainit d40 -m d41
5) re-attach second submirror
# metattach d40 d42
6) mount metadevice
# mount /raid
7) grow the filesystem
# growfs -M /raid /dev/md/rdsk/d40
This was the only way we knew how to fix the mirror issue. If anybody else has experienced this and know of a
better way to resolve the problem when the size of your metadevice mirror does not match the submirrors, it would
be greatly appreciated.
Jojo Jaballas
