Netbackup & NDMP Performance Problem

netapp_fh · ‎2010-09-23

hello,

we are running Netbackup 7.0 and a 3140 active/active cluster with 1 DS14MK2 (sata 1tb disks) / shelf whichi we want to backup with ndmp.

we do have some volumes with 2m to 7m files on them. it doesnt matter which volume we backup, we do not achieve more than 40MB/sec during the backup (sometimes 42 MB/sec )

we backup directly to disk so its not a tape issue. server is a sun x4170 with 2 quad core xeons and 4 gb o ram. attached is a infotrend storage with 12x1tb disk (2 hotspares).

is there a bandwith limit on the netapp according to ndmp traffic? any ideas where this low bandwith is coming from?

thanks for any replies

-andy

columbus_admin · ‎2010-09-27

Hi Andy,

You are most likely running into a common issue that backups of millions small files seem to present. Even going to disk, reading and preparing millions of small files takes a toll on bandwidth. NDMP builds the image, then streams it to the storage device. It is not an NDMP backup issue as I have seen 75MB/sec and higher throughput on direct attached tape.

Now the really horrible way to do this is to backup farther down. For simplicity, if you have 12 qtrees, and 4 DSUs....Allow multistreaming, make sure you have 3 or more jobs allowed per policy, and after every 4th qtree, put in a new directive - NEW_STREAM. What this will do is have Netbackup break the qtrees up into three jobs, with 4 qtrees each. This will break the millions of files up into more streams and should hopefully allow you to get multiple streams at that 40MB mark.

Unfortunately when dealing with millions of files, your inode traversal is a factor as well, there are tons of variables, down to file types(different amounts of meta-data), inode fragmentation, etc.

You are on NBU 7, it will do multistream and now even multiplex on NDMP(though I don't know how, that isn't part of NDMP until version 5! Some internal translation in NBU I assume.) All you are interested in while using DSUs is multistream.

netapp_fh · ‎2010-09-27

hello columbus,

thanks for your input. seems like we do have really a problem i was not very specific with our setup. we have qtress with more than 7m files...

i will do some tests with smaller qtress and only a few 100k files in it. then i should have higher throughput?

so i understand, that until the dump really starts has to do with the nr. of files. so if i understand right, then the nr. of files does not affect the backup speed? then the question is.. why i am no getting more than 40 MB/sec..

seems like further testing is needed...

columbus_admin · ‎2010-09-28

Sorry Andy,

Caught up in some things yesterday and didn't get a chance to respond. Unfortunately not, remember that you still have to "prepare" the files for the NDMP stream, build the stream, and send it. In this case it would be like needing FlashBackup or Advanced Client for a traditional NBU client. It is constantly speeding up and slowing down to build the stream, the same thing that would cause a tape drive to "shoe shine".

Is there anyway someone could script up a gzip of the files for you to backup? If you have a lot of files with similar data and formatting, you may be able to get a really good compression out of it and make some larger files that would back up faster.

** I would NOT recommend what is below, but it may fit your needs in this specific situation **

You could try disabling DAR, but while that may speed up your backup, your restore would be a nightmare. Single file restores are really only made usable by running with DAR enabled. With DAR disabled, NBU restores the entire NDMP dump until it gets to the file you need, then cleans up behind itself. That means you basically need 100% of your largest qtree free at all times, and it take about 10x time to restore single files.

-Scott