Backing up nightly SMVI Snapshots

[jsuttoncs]

I am attempting to backup a nightly SMVI snapshot to tape for longer term retention using NetBackup 6.5.3.1 and NDMP v4.  Obviously, if I had a choice, I would recommend that the customer use SnapMirror and SnapVault, but they have no off-site NetApp storage systems, so they want to backup one SMVI snapshot per day and send those off-site for a certain period of time just in case.  Here are some of the specifics:    

• The customer has three volumes that are hold NFS datastore for VMware
• The customer is using SMVI to take snapshots of these datastores every 4 hours and keeping those for a few days
• The customer would like to backup at least one SMVI snapshot per datastore to tape per day for longer-term retention
• The customer is using an automatic snapshot schedule through SMVI, in other words, no script is generating the snapshots

The main problem I am having is that SMVI does not use a static name for snapshots.  So, even if I call the snapshot NBU-Datastore-Backup in SMVI, SMVI actually names the snapshot smvi_NBU-Datastore-Backup_<date><time>_<GUID>.  Even if I only take one snapshot per day, overwriting the previous days snapshot, the name is different since the date/time and GUID are appended to the snapshot name.

This makes backing up the snapshot via NDMP tricky because backup policies expect a fixed name and NDMP does not support wildcard characters.  So, I can't say backup /vol/vol1/.snapshot/smvi_NBU-Datastore-Backup_*, for instance.  This is what is causing me BIG headaches.  Without being able to use wildcards, how can I do this easily?

Here are a few possibilities I thought of:

• Create a script that wakes up after each datastore snapshot is taken.  Have the script determine the name of the snapshot and modify the NetBackup policy through bppllist with the exact name of the snapshot.  This can be a pain to maintain for the customer.  If they add a datastore, they have to match it up with a NetBackup policy.
• Create a script that creates the snapshot through the SMVI CLI, grabs the name, runs the bpbackup commands, deletes the snapshot.  I don't know if this will actually work as using bpbackup is usually intended to run on a client.  If it is used from the context of a Master/Media server, I believe you have to have a policy configured, which leads me back to the first solution.
• Create a script that creates a snapshot for each datastore through SMVI CLI, grabs the name, then flexclones the snapshot.  NetBackup would use the name of the flexclone as the volume to backup.  This seems the cleanest way.
• Mount the snapshot to another system via SMVI CLI.  Unfortunately, these are NFS datastores and this would drastically slow down data transfer.  The Master/Media server and NetApps are Fibre Channel attached.

Any other suggestions?  I can't believe it is this difficult to backup an SMVI snapshot to tape.  I must be missing something.

[rmharwood]

You're not missing anything and this has been discussed before. I mentioned it some time back but I cannot find my original post.

I believe the ability to have a "predictable" snapshot name is a feature penned for version 2.0 of SMVI.

Richard

[rmharwood]

I found it:

http://communities.netapp.com/thread/1977

[jsuttoncs]

Well, I can get the the names of the snapshots from smvi backup list.  The problem is I need a static name to use with NetBackup.  I thought about renaming the snapshot before running the backup, running the backup, then renaming it to the original name using NetBackup ndmp_start and ndmp_end_notify scripts.  The problem is, what happens if the backup or script aborts for whatever reason?  Then, the snapshot may not get renamed back to its original name and that probably will screw up SMVI.

Then, I thought, why not clone the volume using the latest SMVI snapshot?  That could work.  In other words, when the backup kicks off, the ndmp_start_notify script gets called, that the script figures out the latest SMVI snapshot and clones the volume with that snapshot.  Then, NetBackup backs up the clone, which is always named the same thing for each volume.  Once the backup completes, the ndmp_end_notify script is called, and the clone is removed.  If something the aborts in anywhere, the clone remains, but the next time the backup starts, the ndmp_start_notify script takes cares of removing the clone.

Comments???  Again, the problem is that NetBackup needs a static name to backup.  And, since NDMP doesn't support wildcards, I either have to change the snapshot name, flexclone the volume with a name known by NetBackup, or update the NetBackup policy everytime with the name of the latest snapshot.

[rmharwood]

Flexclone of the volume(/snapshot) sounds like a good idea and might be safer than trying to play nice with SMVI. Let us know how this plays out. We don't currently use SMVI (VIBE works fine for now) but may (have to) one day in the future.

Cheers,

Richard

[jsuttoncs]

No problem.  I will keep you posted ....

[fariborz_behrouzi]

Hello ,

Actually I am doing exactly the same thing and decided to not interfer with SMVI at all . I thought if a backup fails , Storage admin has to clean up the clone and snapshot and found it easier to change the NBU backup selection list with lastest snapshot name.

Currently I am backing up all snapshots and observing a slow perfromance , if I backup volumes it backs up data with 70-90 MB/s and when NBU backs up  snapshot network throughput is  about 20-40 on average. I am using remote NDMP so data with travel back to media server and goes to disk storage.

I would be grateful if let me know what  backup perfromance is ?

Best regards

Fariborz

[jsuttoncs]

I will be performing some tests next week with the customer and will let you know the speeds I get.  Here is what I am proposing, so let me know how close this is to what you are doing:

• The datastore is /vol/DataStore1
• The customer takes one daily snapshot and an hourly snapshot every 4 hours via SMVI
• Daily snapshots are kept for 7 days, while hourly snapshots are kept for 2 days
• The daily snapshots are called DataStore1-Daily and the hourly snapshots are called Datastore1-Hourly
• The NetBackup policy backs up /vol/Datastore1_Backup, which is a flex clone based on the latest daily snapshot
• When the NetBackup policy kicks off at night, the ndmp_start_notify.cmd script calls smvi backup list and determines the latest snapshot, ssh's over to the NetApp and deletes any flex clone named Datastore1_Backup then creates a flex clone based on the latest snapshot
• NetBackup then backs up the flex clone
• Once the flex clone is backed up, NetBackup calls ndmp_end_notify.cmd, which ssh's over to the NetApp and delete the flex clone

Is this the way you are doing it or are you doing something different?  For insance, let's say that the latest SMVI snapshot is 'smvi_backup_Datastore-Daily_200905072330_<GUID>', then the ndmp_start_notify.cmd file ssh's over to the netbackup and runs 'vol clone create Datastore1_Backup -b Datastore1 smvi_backup_Datastore-Daily_200905072330_<GUID>', NetBackup backs up /vol/Datastore1_Backup since that is what is listed in the policies file list, and finally once the backup is complete, the ndmp_end_notify.cmd script deletes Datastore1_Backup via ssh.

What do you think???  Do you have a better way???

[rmatt]

I think the FlexClone option is a good way to go.  You can use 'snap list <volname>' if you want to hardcode a solution and simply grab the most recent snapshot of the volume (presuming you aren't doing scheduled snapshots in combination with the SMVI snapshots).  With that snapshot, make a FlexClone with a consistent name for NDMP backups, and when you're done, destroy the FlexClone.  That way you don't rename the SMVI snapshots and you can use consistent naming conventions in your NetBackup path.

Alternately, you can use SV-SMVI (in the NTN Communities section) to get the most recent SMVI backup information (snapshot name and path) and perform the FlexClone operation in your own script.  Use the -list and either -path or -oneline and then take the snapshot name and volume name to make your FlexClone.  If you know the volume name and don't want to parse the -path or -oneline output, just take the volume name you already know and use -path.  Either way a FlexClone lets you make a consistent path if it has to be the same each time.

If you simply want to take the path and back it up, -list -path is a great way to do it so you don't need to make a FlexClone each time, but if you do that, echo that to a file and simply point your NDMP script to get the path from that file.  Then you don't need FlexClone.  Or, tweak your ndmp_start_notify.cmd script directly to use the output for the full path.

[fariborz_behrouzi]

They way I did it was  a bit different , I was thinking of creating flexclone but it becomes more problematic if something happends to backup script or server .

Snapshot for clone  will stay there and SMVI can not clean up its own snapshots and worse case sernario all VMs will get frozen due to  lack of space on volume ( because of  growing snapshot).

My approach was to add  the volume names  in the backup selection.  when scheduled script start , it will get  the list of snapshot for each volume and based of date, will  select the latest respective snapshot,modifies the backup selection and initiate a manual backup , for example :

for /vol/vmwprod1  volume , script will list the snap shots

  %/used       %/total  date          name
----------  ----------  ------------  --------
  2% ( 2%)    1% ( 1%)  May 07 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090507002200_cc237bd4-1f87-4283-8838-62188c5de55c_rnas7_prod1
  4% ( 2%)    1% ( 1%)  May 06 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090506002200_a66e4e08-48b5-4fd0-9e76-25de25233a3a_rnas7_prod1
  7% ( 3%)    2% ( 1%)  May 05 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090505002200_75778281-1128-4e56-8c49-2ad496c109ae_rnas7_prod1
10% ( 3%)    3% ( 1%)  May 04 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090504002200_6825c5a3-f176-466a-ad56-edf06190f0e4_rnas7_prod1
11% ( 2%)    3% ( 0%)  May 03 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090503002200_002c45f0-8a8c-481f-a150-41e9618ad1e4_rnas7_prod1
13% ( 2%)    4% ( 0%)  May 02 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090502002200_a9f43022-1db4-47da-9265-88c5137cde3c_rnas7_prod1
59% (57%)   36% (33%)  May 01 00:20  smvi_backup_DailyBackup_rnas7_prod1_20090501002200_f7e4e550-7206-4262-8dc1-733a54d89e95_rnas7_prod1

and based on the date (2009 05 07 ) will select:

smvi_backup_DailyBackup_rnas7_prod1_20090507002200

script then does the following:

bpplinclude POLICY -modify IncludePath NewTargetPath

(you can have many volume in the backup selection and  then initiate a backup )

bpbackup -i  -p POLICY -s SCHEDULE

if snapshot is not there or there is any problem , it will send an email to backup admin.

It won't change any thing in the Filer just ssh to read the snap list.

So far it is working and except for network throughput every thing works .

I hope  it helps and please let me know if you need more details.

[jsuttoncs]

Yep, that is another way to go.  Maybe I will do that then.  I think I posted that as one of the possiblities as well.  But, you are correct, creating a flex clone can leave things out there IF the backup or script gets terminated for whatever reason.  Good point.  I just wish they had a better way to do this.  Thanks for the information.  I will write back with the performance test results.  Others have suggested VIBE as well.  This seems simpler.

[rmatt]

This works as well -- if you are worried from a clean-up perspective, you just make sure your script cleans up before it starts the backup, removes any remaining clone, etc.  This isn't terribly difficult to do and would be the same steps in your post-backup operation.  That way any interrupted backup gets cleaned up before a new backup starts (just in case).  You can do this for the FlexClone example, custom ndmp_start_notify.cmd script or anything you do.

I did write a script for someone to do a backup without NDMP but rather host-based backups.  I took SV-SMVI with the -list -path options, and simply added an /etc/exports entry to the snapshot path using an "actual" pathname so it was consistent each time.  Like I just described, if the previous entry was there, we removed that exports entry and then added this new one based on the path to the snapshot.

So the export would be /vol/<volname>/.snapshot/<snapshotofmostrecentSMVIbackup>, and the actual pathname would be /myBackup.

Just another thought if you want to do host-based backups (rather than NDMP -- but recognize it's likely much slower).

Thanks,

--Matt