Data Backup and Recovery
Data Backup and Recovery
Hi all,
A question regarding verify passes on NDMP back-ups.
We're currently performing weekly back-ups of our FAS2050 to tape using Backup Exec 12.5 + NDMP option, with the tape drive attached to the Backup Exec media server (rather than the filer itself) as we're also backing up non-filer data.
The back-up itself executes, and we then run a verify pass to verify the data on the tapes.
We are experiencing the sequential-reads-are-very-slow issue in our release of Data ONTAP (7.2.4L1), which is resulting in the backup pass only running at 15MB/s or so. Annoying, but we know a fix is in the works. I noticed, however, that the verify pass runs at closer to 50MB/s, and was investigating why this is as I would have expected it to be similarly slow.
Looking at sysstat on the filer while the verify pass is running, a large amount of data is being received at the filer, whereas I would have expected it to be sending a large amount of data. I can only assume this is because the verify operation sends the data from tape -> server -> filer and the verify happens at the filer, rather than than data going from tape -> server and filer -> server and the verify happening at the server as I was expecting? This would help to explain the better performance during the verify pass as well.
What I can't expain, however, is this: during the verify pass the filer is receiving data at 40-60MB/s on it's network interfaces. However, the Disk Read stats in sysstat are much lower: 1MB/s - 5MB/s (which is our "normal" disk read load). This makes me wonder if the data from the back-up isn't actually being verified?
Can anyone explain this? Is the filer only pretending to verify the data (so it's just wasted effort to run a verify pass)? Or does the filer verify the back-up data based on checksums/metadata rather than byte-by-byte, which therefore would require very little in the way of actual disk reads?
Thanks,
Matt
I am not sure what is going on with the verification process as I am just an end user however you could try and troubleshot the issue yourself.
If you do a dump to null of the volume on the filer it will tell you the max speed the data will come off the disks. This is because it does not write the data anywhere and just reads it fast as possible.
You can turn on NDMP debugging while the verification is running and see what is happening at a low level. Level 70 creates loads of data in a very short time so don't forget to turn it off. Record console output to a log file.
ndmpd debug 70 {max level}
ndmpd debug 00
run perfstat on thebackup host and point it at your filer. This will tell you everything you need to know and loads do don't....
Hope this helps, or at least until someone with the solution posts.
Merry Christmas dude
Brendon
Out of Office e-mail got posted automatically, removing...
Message was edited by: Mathew Kilham
Did you post thelast email or did OoO post for you?
If we get two OoO I guess we know the answer....
OoO posted it for me.
You can turn on NDMP debugging while the verification is running and see what is happening at a low level. Level 70 creates loads of data in a very short time so don't forget to turn it off. Record console output to a log file.
>
Hope this helps, or at least until someone with the solution posts.
Thanks for the suggestion - I've just sent my question to NetApp support, if they don't come back with some useful info I will try your idea.
Merry Christmas to you too!
Hi Mathew,
have you got any response form NetApp support? I am here facing the same problem when backing-up Non-filer data to filer attached library.
we are able to get Max 15-18MB backup throughput with symantec Backup Exec 12.5.
Plz help us if you have any idea...
---Vikram
Hello,
Is there any progress in this case? Because unfortunately we have the same problem. We using a Netapp FAS2020C and Backup Exec 12.5 and the library is connected to one of the heads. We first using Data ONTAP version 7.2.5.1, but because of the article: http://www.netapp.com/us/solutions/a-z/backup-to-tape/backup-to-tape-ndmp.html(Backup Exec 12,5 Requires Data ONTAP 7.3)
I upgrade to Data ONTAP 7.3.1.1. But the backup speed is still the same, around the 18MB.
Sander