Tips for tuning imsbackup, especially with Legato Networker
We've been working really hard to improve our backup performance. I
thought I'd capture what we've done so far. We've been able to nearly
double our throughput to 35-37MB/second. We were well below 20MB/sec a
few months ago
1) If you're running Sun Cluster, do not use the "Global" (aka pxfs)
filesystem for mail store partitions. If you do, you'll be CPU-bound.
2) Set the noatime option on your mail spool partitions. This will
eliminate a write to update the access time at every read and was a win
for us.
3) If you are using Legato Networker, or any other backup software that
uses imsbackup in a pipeline make sure you apply the fifofs (pipe)
performance patch. It's patch 117471 for Solaris 9 and patch 109454 for
Solaris 8. This patch improved our backup performance by nearly 10MB/sec.
4) If you are using Legato Networker (version 7.1) and doing
disk-to-disk backups, do not use the "Advanced File Type" device. This
device type uses the nsrfile packet size to write to disk, so disk write
block sizes end up between 2KB and 8KB, which will totally kill
performance. We don't know if the same problem exists in later versions
of Networker. This improved our tape cloning performance from about
20-24MB/sec to 50-70MB sec. We also found that running one nsrmmd
process per CPU to maximize performance.
5) Consider increasing the imsbackup blocksize from the default of 10KB.
My testing showed a sweet spot at 64KB (-b 128). A block size of 128KB
(-b 256) was sometimes faster, and sometimes slower. If you're using
imsasm, you'd make the change in /usr/lib/nsr/imsasm, and you need to
change it for imsrestore as well. Here's a table of my test results:
blocksizeSysUserWall Clock
default15.1637.6907:22.3
default14.1536.0706:26.2
default14.5637.5706:59.9
default14.8535.8607:32.9
default14.2636.8806:00.6
default14.3737.3505:11.1
-b 64 13.2834.2805:18.6
-b 64 12.1432.6904:51.7
-b 64 12.2933.5904:26.7
-b 64 12.1234.3804:29.9
-b 64 12.7734.2704:27.7
-b 64 12.5332.8404:34.4
-b 128 12.5931.0604:43.1
-b 128 12.7134.8204:16.4
-b 128 12.4227.1204:13.5
-b 128 11.8732.5404:35.6
-b 128 11.9833.7304:31.6
-b 128 11.4534.6504:13.8
-b 256 11.7226.6304:12.0
-b 256 11.8925.0904:02.2
-b 256 12.6827.2304:29.1
-b 256 11.9628.9504:09.9
-b 256 12.4630.605:04.4
-b 256 11.8626.7905:13.6
-b 512 11.9326.3804:16.4
-b 512 11.6927.8104:26.2
-b 512 12.4426.1204:42.7
-b 512 11.3726.2304:24.4
-b 512 12.0434.0904:51.5
-b 512 11.5627.0304:27.3
Other items we're looking at are:
1) adding more spindles to our disk-to-disk array to improve write performance
2) Use jumbo frames to improve network performance and reduce CPU overhead
If anyone else has tips for improving backup performance, I'd love to
hear them!
-jeff
[3297 byte] By [
jwasilkoa] at [2007-11-27 8:29:08]
# 1
Hi Jeff,
Thanks for the excellent summary of your results. Given that you have access to production systems your own experience reflecting real-world-results is very useful.
>> If anyone else has tips for improving backup performance, I'd love to hear them!
There are the default networking tuning tips (for both mail-store and backup server) but I suppose you have already gone through them e.g.
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 65536
/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 65536
I suppose the question at this point is.. what is the current bottleneck, CPU/Network/I/O?
Depending on which one is slowing things down will determine where the effort needs to be put.
Regards,
Shane.
# 2
> Hi Jeff,
>
> Thanks for the excellent summary of your results.
> Given that you have access to production systems your
> own experience reflecting real-world-results is very
> useful.
You're welcome! I hope it's useful to other folks.
> /usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 65536
> /usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 65536
Yup, we've got those set.
> I suppose the question at this point is.. what is the
> current bottleneck, CPU/Network/I/O?
I think we're close to being CPU-bound on both the JES host and the Legato side. I'm curious how much jumbo frames would help us in terms of reducing kernel time.
The backup server is an easy upgrade, since it's a 280R, but the JES cluster is a pair of 1280s (8CPU). Adding CPUs to those are expensive....
If you'd like to take a look at the systems performance, http://orca.usg.tufts.edu/orcallator/o_oak-daily.html is the active node in the cluster and
http://orca.usg.tufts.edu/orcallator/o_ash-daily.html is the inactive node (but was active until I did the pipe patch at 17:00 today, so it's useful to use for history.
http://orca.usg.tufts.edu/orcallator/o_alexandrite-daily.html is the backup server
Note that the disk reads/write rate and ops/sec are inflated because orca is counting the disksuite devices as well as the regular devices, so everything is roughly double.
-j
# 3
Hi Jeff,
> I suppose the question at this point is.. what is
> the
> current bottleneck, CPU/Network/I/O?
>
> I think we're close to being CPU-bound on both the
> JES host and the Legato side. I'm curious how much
> jumbo frames would help us in terms of reducing
> kernel time.
http://www.psc.edu/~mathis/MTU/AlteonExtendedFrames_W0601.pdf
They talk about a large drop in CPU utilisation (around 50%) *results may vary
To get the best throughput you would want end-to-end jumbo frames which mean you may need to tweak/upgrade switches/routers between the two systems.
> The backup server is an easy upgrade, since it's a
> 280R, but the JES cluster is a pair of 1280s (8CPU).
> Adding CPUs to those are expensive....
Agreed. If you can get the JES cluster to be the one waiting for the backup server, definitely the best option. I take it that neither system is running Solaris 10 (so you could use dtrace to probe into the underlying high CPU use?).
> If you'd like to take a look at the systems
> performance,
> http://orca.usg.tufts.edu/orcallator/o_oak-daily.html
> is the active node in the cluster and
> http://orca.usg.tufts.edu/orcallator/o_ash-daily.html
> is the inactive node (but was active until I did the
> pipe patch at 17:00 today, so it's useful to use for
> history.
One thing I noticed is the 'new process' spawn rate is pretty high (~200/s). Any idea what is causing this?
Also have you considered binding the imsbackup processes to a specific cpu, to stop the processes being bounced around? May not make much difference but something to try.
Regards,
Shane
# 4
>
> http://www.psc.edu/~mathis/MTU/AlteonExtendedFrames_W0
> 601.pdf
>
> They talk about a large drop in CPU utilisation
> (around 50%) *results may vary
Thanks for the pointer. It'd be nice to get something close to 50% :-)
> o get the best throughput you would want end-to-end
> jumbo frames which mean you may need to tweak/upgrade
> switches/routers between the two systems.
We're set for that. I've been working on setting up a separate backup VLAN that only carries jumbo frames.
> Agreed. If you can get the JES cluster to be the one
> waiting for the backup server, definitely the best
> option. I take it that neither system is running
> Solaris 10 (so you could use dtrace to probe into the
> underlying high CPU use?).
Yeah, we're still at Solaris 9. dtrace would have made a lot of this easier!
> One thing I noticed is the 'new process' spawn rate
> is pretty high (~200/s). Any idea what is causing
> this?
I think it's a SAN health monitoring script called lunmapper we got from Sun. It's written in ksh, and is forking lots of stuff (like echo and expr). We run it out of our monitoring system every 5 minutes (I think). I'll look into running it less often.
> Also have you considered binding the imsbackup
> processes to a specific cpu, to stop the processes
> being bounced around? May not make much difference
> but something to try.
Hmm. Hadn't thought of that. We actually run 25 backup jobs at once at peak, so there's more processes than CPUs. Not sure it'd help much then.
Thanks for the tips!
-j
