making (and testing) a disaster recovery backup

I need a DR backup for our NetBackup server. The server is a V440 running Solaris 9. It has four 73 GB internal drives setup as two mirrored pairs. There is some space on our SAN (6130) also. NBU uses a Dell PowerValt containing 6 LTO2 tape drvies. The V440 has a DVD drive. I have a V490 with 2 146GB internal disk drives, access to the same tape library and a DVD drive that I can use to test the DR backup.

We are using NBU to backup the V440 so I'm going to say that I do not need another backup of the space on the SAN. I expect to use NBU to restore those file systems in a DR situation.

Here's a current df -h:

$df -h

Filesystem sizeused avail capacity Mounted on

/dev/md/dsk/d0 57G2.6G54G5%/

/proc0K0K0K0%/proc

mnttab0K0K0K0%/etc/mnttab

fd0K0K0K0%/dev/fd

/dev/md/dsk/d1 1.9G1.0G869M55%/var

swap14G40K14G1%/var/run

swap512M336K512M1%/tmp

/dev/md/dsk/d10098G14G83G15%/export

/dev/md/dsk/d102246G35G208G15%/nbu/disk

/dev/md/dsk/d1011.1T899G227G80%/nbu/staging

/dev/md/dsk/d5 67G13G54G20%/opt/openv

1. Is ufsdump a good toolforthis type of backup?

2. Must I shutdown the server and boot from a DVD/CD? If yes,

1. What is involved with accessing a tape drive over the SAN? (Willthis be a problem when booting from DVD/CD?)

2. What commands are needed to mount the file systems?

3. /dev/md/dsk/d100, /dev/md/dsk/d101 and /dev/md/dsk/d102 are on the SAN - I am not planning to back them up using ufsdump. That leaves /dev/md/dsk/d0, dev/md/dsk/d1 and dev/md/dsk/d5 that Ido need to backup - right?

4. Anything special needed to stack the three file systems on one tape?

I'm sure other questions will come up as I assemble the actual steps and commands.

Thanks for any ideas,

Glen Gunselman

[1989 byte] By [sysglen] at [2007-11-26 6:20:04]
# 1
I did not receive any input from the forum on this topic. Is there a better forum for this type of topic or is the idea of making system backups just not something we talk about?hoping you have a better day with your question,Glen
sysglen at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 2

Since you're on Netbackup, have you considered Veritas Bare Metal Restore.

What are your DR plans if the SAN goes south? Think hurricane Katrina type damage.

Setting up stand by servers at another location while expensive, may or may not be what you need.

What are the parameters around your DR needs?

alan

alan_pae at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 3

At this point my DR plans consider only loss of data - not hardware. (I do have some test server hardware but only it's internal storage.) I need a way to recover the Netbackup management server's environment to the point that I can use Netbackup.

I have heard of Veritas Bare Metal Restore but our Sun Vendor made no effort to sell it when we were buying Netbackup.

Do you use Bare Metal Restore? Do you shutdown the server to single user mode, or do you boot from an alternate file system (in the (IBM) mainframe world we can boot (IPL) from tape - can Solaris boot from a tape?)

If I restore the V440 backup to a V490 can I expect to run Netbackup using the current license key?

--

Here's the commands that I have so far. (This was done with the server up, so it's not a good backup. All the commands worked except the tape unload at the end. I had to manually (using the controll panel on the tape library) remove the tape. Also, I used trail and error to determine which Solaris tape drive the tape was mounted in.):

honeydew ~ gunselmg $su

Password:

# echo "m p1 d1" | /usr/openv/volmgr/bin/tldtest -r /dev/sg/c0tw200100308c03992bl1 -d1 /dev/rmt/5cbn -d2 /dev/rmt/4cbn -d3 /dev/rmt/3cbn -d4 /dev/rmt/2cbn -d5 /dev/rmt/1cbn -d6 /dev/rmt/0cbn

Opening /dev/sg/c0tw200100308c03992bl1

MODE_SENSE complete

Enter tld commands (? returns help information)

Initiating MOVE_MEDIUM from address 16 to 256

MOVE_MEDIUM complete

#

# mt -f /dev/rmt/0 status

/dev/rmt/0: no tape loaded or drive offline

# bash

honeydew ~ root #mt -f /dev/rmt/1 status

/dev/rmt/1: no tape loaded or drive offline

honeydew ~ root #mt -f /dev/rmt/2 status

/dev/rmt/2: no tape loaded or drive offline

honeydew ~ root #mt -f /dev/rmt/3 status

/dev/rmt/3: no tape loaded or drive offline

honeydew ~ root #mt -f /dev/rmt/4 status

/dev/rmt/4: no tape loaded or drive offline

honeydew ~ root #mt -f /dev/rmt/5 status

IBM Ultrium Gen 2 LTO tape drive:

sense key(0x6)= Unit Attentionresidual= 0retries= 0

file no= 0block no= 0

honeydew ~ root #mt -f /dev/rmt/6 status

/dev/rmt/6: No such file or directory

honeydew ~ root #mt -f /dev/rmt/5 rewind

honeydew ~ root #ufsdump 0ucf /dev/rmt/5n /

DUMP: Date of this level 0 dump: Wed Apr 12 15:24:29 2006

DUMP: Date of last level 0 dump: the epoch

DUMP: Dumping /dev/md/rdsk/d0 (honeydew:/) to /dev/rmt/5n.

DUMP: Mapping (Pass I) [regular files]

DUMP: Mapping (Pass II) [directories]

DUMP: Writing 63 Kilobyte records

DUMP: Estimated 5391346 blocks (2632.49MB).

DUMP: Dumping (Pass III) [directories]

DUMP: Dumping (Pass IV) [regular files]

DUMP: 5391286 blocks (2632.46MB) on 1 volume at 13474 KB/sec

DUMP: DUMP IS DONE

DUMP: Level 0 dump on Wed Apr 12 15:24:29 2006

honeydew ~ root #ufsdump 0ucf /dev/rmt/5n /var

DUMP: Date of this level 0 dump: Wed Apr 12 15:28:33 2006

DUMP: Date of last level 0 dump: the epoch

DUMP: Dumping /dev/md/rdsk/d1 (honeydew:/var) to /dev/rmt/5n.

DUMP: Mapping (Pass I) [regular files]

DUMP: Mapping (Pass II) [directories]

DUMP: Writing 63 Kilobyte records

DUMP: Estimated 2158000 blocks (1053.71MB).

DUMP: Dumping (Pass III) [directories]

DUMP: Dumping (Pass IV) [regular files]

DUMP: 2157874 blocks (1053.65MB) on 1 volume at 12636 KB/sec

DUMP: DUMP IS DONE

DUMP: Level 0 dump on Wed Apr 12 15:28:33 2006

honeydew ~ root #ufsdump 0ucf /dev/rmt/5n /opt/openv

DUMP: Date of this level 0 dump: Wed Apr 12 15:30:31 2006

DUMP: Date of last level 0 dump: the epoch

DUMP: Dumping /dev/md/rdsk/d5 (honeydew:/opt/openv) to /dev/rmt/5n.

DUMP: Mapping (Pass I) [regular files]

DUMP: Mapping (Pass II) [directories]

DUMP: Writing 63 Kilobyte records

DUMP: Estimated 26212514 blocks (12799.08MB).

DUMP: Dumping (Pass III) [directories]

DUMP: Dumping (Pass IV) [regular files]

DUMP: 26212408 blocks (12799.03MB) on 1 volume at 33370 KB/sec

DUMP: DUMP IS DONE

DUMP: Level 0 dump on Wed Apr 12 15:30:31 2006

honeydew ~ root #date

Wed Apr 12 15:37:34 CDT 2006

honeydew ~ root #echo "m d1 p1" | /usr/openv/volmgr/bin/tldtest -r /dev/sg/c0tw200100308c03992bl1 -d1 /dev/rmt/5cbn -d2 /dev/rmt/4cbn -d3 /dev/rmt/3cbn -d4 /dev/rmt/2cbn -d5 /dev/rmt/1cbn -d6 /dev/rmt/0cbn

Opening /dev/sg/c0tw200100308c03992bl1

MODE_SENSE complete

Enter tld commands (? returns help information)

Initiating MOVE_MEDIUM from address 256 to 16

move_medium failed

sense key = 0x5, asc = 0x53, ascq = 0x1, UNLOAD TAPE FAILURE

honeydew ~ root #mt -f /dev/rmt/5 statusIBM Ultrium Gen 2 LTO tape drive:

sense key(0x0)= No Additional Senseresidual= 0retries= 0

file no= 3block no= 0

honeydew ~ root #mt -f /dev/rmt/5 status

IBM Ultrium Gen 2 LTO tape drive:

sense key(0x0)= No Additional Senseresidual= 0retries= 0

file no= 0block no= 0

honeydew ~ root #mt -f /dev/rmt/5 rewind

honeydew ~ root #echo "m d1 p1" | /usr/openv/volmgr/bin/tldtest -r /dev/sg/c0tw200100308c03992bl1 -d1 /dev/rmt/5cbn -d2 /dev/rmt/4cbn -d3 /dev/rmt/3cbn -d4 /dev/rmt/2cbn -d5 /dev/rmt/1cbn -d6 /dev/rmt/0cbn

Opening /dev/sg/c0tw200100308c03992bl1

MODE_SENSE complete

Enter tld commands (? returns help information)

Initiating MOVE_MEDIUM from address 256 to 16

move_medium failed

sense key = 0x5, asc = 0x53, ascq = 0x1, UNLOAD TAPE FAILURE

honeydew ~ root #mt -f /dev/rmt/5 status

/dev/rmt/5: no tape loaded or drive offline

honeydew ~ gunselmg $

have a good day,

Glen

sysglen at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 4
Forgive how this sounds but it sounds like all you need is a normal backup routine.Information on how to backup and restore Netbackup critical files can be found at support.veritas.comalan
alan_pae at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 5
I'm not sure what is a normal backup routine in the Solaris world. Is shutting down to single user mode and running ufsdump considered a normal backup? I've seen several comments saying that you can not really backup a mounted file system.have a good day,Glen
sysglen at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 6

Technically, you can backup a mounted file system, but if something is constantly writing to it, you'll never finish or will corrupt your backup.

Based on the responses, I think the issue here is that some admins( myself included) see backups and DR as 2 separate issues. It goes something like "Backups are used when a file is lost. DR is used when there's a hole in the ground where the data center used to be".

There are 100 ways in which you can backup your data, depending on what kind of data you are handling, how much data you are handling, how fast it needs to back up, how fast you can restore it, how often you collect new data, plainly which process/application you happen to like better, among other things... so a definitive answer will depend on your specific scenario.

Now, if I understand correctly, what you want to backup is your backup server itself, right?

If so, without knowing any more specifics other than what you posted, what I would do for backup, I'd back up the file system using a flash archive and copy the configuration files somewhere, being either tape, some other server, etc...

As for DR, I'd have the exact same server and config in another state and copy my configuration files every so often from production to standby. If my production is in, say, Florida, I'd put my standby in somewhere in the midwest. If my server is in Oklahoma, I'd put my standby in Seattle...

But again, it really depends on your specific situation and what process you feel confortable with.

Codename47 at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 7

Yes, I need to backup the backup server. I'm want to create a backup that I can store offsite and that could be restored to another server should the need arise.

My current resources include an extra tape, some space in another building, the production hardware (includes server - running Solaris 9 - containing the files w/access to tape drives) and some available time on Friday evenings while I'm waiting/running backups on other servers, and some Sun doc's (Intermediate System Administration for the Solaris 9 Operating Environment Volume 2 of 2 [from class SA-239] - I'm looking at page 16-21) using ufsdump to backup / (root).

I know a little about jumpstart. I believe to use jumpstart I need either another server configured as a jumpstart server or I would need some way to burn a DVD (although a single DVD could not contain all the files, at this point). I don't see how I could produce a flar that I could transport to another building more conveniently than a single tape using ufsdump.

Is there something about creating flars that makes the process more reliable than ufsdump?

Thanks for the input,

Glen

sysglen at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 8

> My current resources include an extra tape, some

> space in another building, the production hardware

> (includes server - running Solaris 9 - containing the

> files w/access to tape drives) and some available

> time on Friday evenings while I'm waiting/running

> backups on other servers, and some Sun doc's

> (Intermediate System Administration for the Solaris 9

> Operating Environment Volume 2 of 2 [from class

> SA-239] - I'm looking at page 16-21) using ufsdump to

> backup / (root).

What we need from you.

List of all hardware, production, spares, bone pile etc....

Backup devices and backup media and quantity

What process are running on the production server and what files

are critical to your business. We don't need to know the nitty gritty

just the high level stuff. Like we run Oracle, or we run Peoplesoft and

the following file systems need to be backed up, what's there isn't really important I just need to get them backed up.

You've mentioned NetBackup, Jumpstart, Flash, ufsdump. Pick one for us so we aren't just taking pot shots at what you're trying to do. Since you've hopefully paid for NetBackup why are you looking at jumpstart, flash, etc...?

You have space in another building, ok, but is there any hardware to go with that space? Is there power, cooling, raised floors, rack space, etc...?

Do you have the enough hardware where you can take production backups and then restore them onto another set of boxes to see what happens when you restore?

alan

alan_pae at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 9

Well, as far as and advantage between one and the other, I guess that to me

the advantages are that I don't need an extra piece of equipment and media

(tape and tape drive), I can transfer the flar electronically anywhere I want,

and make as many copies in seconds as I want, I can install it from anywhere

to anywhere...I may need a jumpstart/boot box, but any box can do this...

There's nothing wrong with using ufsdump; like I said, I think is a matter of

which process you are more comfortable with. These all are just tools and

each individual uses these tools differently. The devil is in how you set the

process. Obviously, you are also limited by what resources are available

to you, and, for what you listed as your resources, ufsdump may be the only

way to go.

As for documentation, you may want to peruse docs.sun.com or

sun.com/blueprints...

Codename47 at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...
# 10

We use flash archives quite extensively. There is an option on the boot cd/dvd to point the install to a flash archive. I would recommend going down to single user mode and getting a good flash archive. We actually took a 36GB flash archive of a loaded V880 and dropped it onto a V120. It actually worked, it was slow, but it worked! Pretty sweet! You can alter profiles and drop it onto any machine of the same arctitecture, which almost anything still running is a sun4u. I would load any additional drivers onto the machine that you my need prior to making the flash archive to include graphics drivers, storage array drivers, and anything that another system may require. Also, make sure the system you are making a flash archive of has the package SUNWnisr installed. You can remove it once you have created your archive, but it needs to be there when the archive is captured or you won't have much luck trying to re-install it.

DISAFSO at 2007-7-6 14:04:18 > top of Java-index,General,Sys Admin Best Practices...