Community

Subscribe
Highlighted
Accepted Solution

NDMP Backup 2MB/min

Hi,

is there a limitation (file count) for NDMP? We see a very very poor Performance with NDMP Backups of a 2040c with BackupExec 2010r3. We have not evaluated a different backup software yet so we are not sure if this is the bottle neck.

The volume we are trying to backup consists of several qtree's and has a total around 16 Mio Files.

Regards,

Oliver

NDMP Backup 2MB/min

I'm assuming the filer is directly connected to the tape device and data flow isn't passing through a media server?

As a start you can issue `sysstat -u 1` from the CLI and check for any obvious bottlenecks i.e. CPU, disk utilisation plus you'll get a clear view of approximate KB/s the filer is writing to tape.

Re: NDMP Backup 2MB/min

The first step dump does is to build full file list; with large number of files it can be quite time consuming. You can test the theoretical dump speed by calling it directly and dumping to NULL:

https://kb.netapp.com/support/index?page=content&id=1011894

Watch how much time each of steps takes. You should also be able to see dump logs in /etc/log/backup.

NDMP Backup 2MB/min

Hi and thanks for the quick responses:

I'm assuming the filer is directly connected to the tape device and data flow isn't passing through a media server?

- Yes it is directly attached to our FAS2040 via FC.

I have now snapmirrored the largest qtree (1.8mio files) to a new volume and started backup exec. Let's see how long it takes. If this is already an issue we have to review the space usage.

Regards,

NDMP Backup 2MB/min

sysstat shows a very high disk usage (80-97%) when running the backup.

NDMP Backup 2MB/min

Ok, once the backup has really started the disk usage drops below 30%. It is just sitting around for about 15 minutes. I guess this is when the list of files to backup is created.

NDMP Backup 2MB/min

We've had problems similar to this when we back up volumes that contain large numbers of files.  It has to crawl the file list before it actually starts backing up, and sometimes that can take hours before it actually starts moving any data.  Commvault supposedly now has a solution that allows it to skip that crawl by using snapshots and only looking at changed data.  Another option that NetApp presented to us is doing a volume based backup.  That avoids the file crawl, but doesn't allow individual file restores... you have to restore the entire volume.  Might be an option if you can use snapshots for individual restores and the volume backup for major disasters.

NDMP Backup 2MB/min

I should also add that this is not a problem that is unique to NetApp.  We had the same issues doing NDMP backups from a Celerra as well.

NDMP Backup 2MB/min

Hi,

thanks for the recommendations. Is there a trial version of commvault available? I had recommended commvault in the past to our boss but we had backup exec for several years now and it is quite ok. I will now evacuate all qtrees wth more than 600.000 files to a new volume and create a separate backup job for it. We have two LTO 5 FC Tape drives, so we can split it up.

NDMP Backup 2MB/min

hi,

You said: "The volume we are trying to backup consists of several qtree's and has a total around 16 Mio Files."

Thats your problem right there. NDMP is not good for lots of small files. Off the top of my head I think NDMP backups

have to 5 phases (or was it 6) and the first 3 ones are just for scanning the file system. Imagine having to scan

that many files on a very small controller. You havent got the CPU to deal with this I reckon, even if you did it would

take ages to back this up.

The way to work around this is to use snap products as backup methology. As mentioned commvault integrates nicely

with netapp snaps.

Cheers,

Eric