2011-07-31 03:23 AM
A while back, when we first started using SMHV, we were stumped as to how we could backup the snapshots from the filer to tape for long-term storage. To solve that problem, we created a nice and effective solution (http://communities.netapp.com/message/22958#22958) which allowed us to backup to tape, as well as backup the individual VHD files without having to backup via NDMP and in the event where a restore would be required, we would have to first restore the entire LUN and only then would we need to restore the VHD from the LUN. We were doing this for over a year and were content with that solution. In the interim, I haven't heard any other methods used by anyone to make us think otherwise.
Recently, our filer has started to experience peaks where the CPU is pegged at 100% (or close to it) for a while, causing our Hyper-V servers to temporary loose connectivity to the filer for a short period of time, which caused all the VMs running on those boxes to crash and restart. Apparently, this is related to the iSCSI Initiator not being able to maintain a connection with the filer while the filer is too busy to respond in due timing. This happens to our 2008 R2 servers. Our 2003 servers, which are running LOB apps, just blue screen, crash, and reboot, where the error shown in debug anlysis points to the iSCSI initiator. (I mention this as a preface although I wonder if anyone else has ever experienced this issue.)
We opened a case with NetApp, and after analyzing perfstat outputs, it was determined that the high CPU utilization is indeed caused by the way we backup our Hyper-V LUNs to tape, with the explanation given that the backup is causing high amounts of "sparse reads", which in turn causes the filer's CPU to burst, which in turn causes our system to crash. :-(
So now we are back at square one, looking for a way to backup SMHV snapshots to tape.
Possible Solution A
The following is my analysis of the options presently available. I would love to hear from other people what they are doing.
Backing up via NDMP (as opposed to backing up individual VM Disk files)
Up until recently we were backing up our the data from our production filer. We have recently purchased a second filer for our DR site and have started to replicate the data over to the remote location. Now we need to select the source of the data we want to backup. Well - that depends on how we do replication.
If we use QTree replication, then:
If we use Volume level replication, then:
As this post gets longer and longer - in general I was wondering, what do people do to solve these issues, and does anyone have any ideas or potentiall solutions?
2011-07-31 11:14 AM
SMHV does not rename the snapshot with _recent suffix, but it does provide the name of the snapshot that was created as part of backup process in $VMSnapshot environment variable. This variable can be checked in the post backup script to get the name of snapshot. Can this be used to solve the problem of knowing which snapshot to transfer / backup?
The backup (the one with _backup suffix) snapshot has two LUNs (one with _exclude and one without). But only one LUN, the one without _exclude contains the auto recovery changes and this is actually a LUN clone. This snapshot only has the auto recovery data. The actual VM data is present in the original LUN in the first (backing) snapshot.
2011-08-03 02:57 AM
We know about the $VMSnapshot already. We use it presently in our old backup solution I wrote about in the post mentioned above. The problem is that when we mounted this snapshot and backed it up file by file, it was causing CPU spikes on our filer.
When we tried to backup via NDMP, we could only select the QTree Level, which forces us to backup both of the LUNs, the clone and the real LUN. Secondly, we didn't know of a simple way to dynamically change the selection list within our backup program to tell it which snapshot path to backup (assuming we knew how via the $VMSnapshot variable). That's two negatives.
Thirdly, with QTree SnapMirror we cannot backup this data on the destination filer since there is no snapshots of similar names, and therefore no consistent data with which to backup. And even if we used Volume SnapMirror to solve this issue, and that would give us similar snapshot names as the source, we are still presented by the first two problems above.
2011-08-26 10:46 AM
It might be overkill for what you need but the Snap Creator framework will take care of problem #3 above. It's a very flexible tool that has options for pre and post snapshot operations, snapvault, renaming based on configurable inputs, etc. They don't spell out SMHV on the Snap Creator main page but that tool has been adding capabilities like wildfire and it might be worth a post to the Snap Creator community to see if it would do the job. From what I have seen it is so flexible that you can write a plugin to do just about anything you need if it doesn't already do it.
There is a white paper on advanced scenarios and Snap Creator here if you are interested...
Oh and did I mention...it's free. :-)