Data Backup and Recovery Discussions

SMVI - backing up VMs with iSCSI LUNS and SnapManager

marcconeley

Is it possible to use SMVI v2 to perform VM consistent snapshots of VMs that have iSCSI LUNs mounted?

I am using vSphere (latest v) and SMVI v2. All my VMs are mounted on an NFS volume on my FAS2020.

All my Exchange, SQL and Sharepoint servers are using SnapManager for backups, and therefore have their data mounted on iSCSI initiator LUNs within the VMs.

Until recently I was creating hourly "non VM consistent" snaps and daily "VM consistent" snaps (according to: http://blogs.netapp.com/virtualization/2009/07/scheduling-smvi.html).

I encountered 2 main problems with this:

1) The daily VM consistent snapshots would always fail on 4-5 of my servers, randomly it seemed. The rest would snapshot OK.

2) I noticed that many of the volumes containing my iSCSI LUNs were rapidly filling up.

It seems that the random server snapshot failures all had 1 thing in common: they all had iSCSI LUNs (why sometimes they worked though, and sometimes not I have no idea!).

It also seems that by performing SMVI "VM consistent" snapshots of these servers, a conflict is caused with SnapManager which also results in the iSCSI LUN (mainly the SnapInfo one in the case of SQL!) being snapped. This happens outside of the control of SnapManager. So in my case, after 1 week my SnapInfo vol reported full - when I checked the snapshots on this vol I could see 7 days worth of SQL snaps (normal) - but also 7 days of additional SMVI snapshots!! (very bad).

This happened on normal VM servers with iSCSI LUNs mapped and my SQL servers.

Since then I read somewhere that VMware do not support VM snapshots of VMs with Microsoft initiated iSCSI LUNs (which rules out half my VMs!) and therefore I've removed these servers from my backup.

Is there any way around this problem?

Longer term my NetApp partner is currently selling me a backup project whereby we will snapshot all VMs and then snapmirror them to a remote site for DR. The problem is that my most important VMs have iSCSI LUNs. My understanding according to the above article is that VM consistent snapshots are important as the VM is quiesced and the snapshots are clean. Therefore my backup idea is not good as basically I'd be mirroring "dirty snapshots" to my remote DR site. Ideally these snapshots should be clean, VM consistent ones (or am I over valuing the importance of this?).


Help appreciated.

//Marc

25 REPLIES 25

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

amritad

Hi

According to VMware KB article #1009073, VMware Tools are unable to create quiesced

snapshots of virtual machines that have NPIV RDM LUNs or Microsoft iSCSI Software Initiator

LUNs mapped to them (this often results in timeout errors during snapshot creation). Therefore,

customers using the Microsoft iSCSI Software Initiator in the guest and running SMVI with VMware

snapshots turned on, which is not recommended, are at high risk of experiencing SMVI backup

failures due to snapshot timeouts caused by the presence of Microsoft iSCSI Software Initiator

LUNs mapped to the virtual machines.

VMware’s general recommendation is to disable both VSS components and the sync driver in VMware Tools

(which translates to turning off VMware snapshots for any SMVI backup jobs that include virtual machines

mapped with Microsoft iSCSI Software Initiator LUNs) in environments that include both Microsoft iSCSI

Software Initiator LUNs in the VM and SMVI, thereby reducing the consistency level of a virtual machine

backup to point-in-time consistency. However, by using SDW/SnapManager to back up the application data on the Microsoft iSCSI Software Initiator LUNs mapped to the virtual machine, the reduction in the data consistency level of the SMVI backup has no effect on the application data.

Another recommendation for these environments is to use physical mode RDM LUNs, instead of Microsoft

iSCSI Software Initiator LUNs, when provisioning storage in order to get the maximum protection level from

the combined SMVI and SDW/SnapManager solution: guest file system consistency for OS images using VSS-assisted SMVI backups, and application-consistent backups and fine-grained recovery for application data using the SnapManager applications.

Regards

Amrita

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

radek_kubka

Hi Amrita,

I must admit I missed this earlier:

VMware Tools are unable to create quiesced snapshots of virtual machines that have NPIV RDM LUNs or Microsoft iSCSI Software Initiator

LUNs mapped to them

[...]

Another recommendation for these environments is to use physical mode RDM LUNs,

In other words, if you run Windows VMs over iSCSI, use SMVI & SnapManager products, you really should have SnapDrive 6.2 in your environment as earlier versions can't use anything else that MS software iSCSI intiator, correct?

Regards,
Radek

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

amritad

Hi Radek,

I'm not sure I got your question. Even previous version of SDW support Microsoft iSCSI software initiator LUNS. With 6.2 we get the pass through disk support.

Regards

Amrita

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

radek_kubka

OK, to make my question complete:

If you use SMVI and would like to have OS-consistent backups of VMs and these VM use some iSCSI RDMs (requiring SnapManager protection), then you need SnapDrive 6.2, as earlier SDW versions require software iSCSI initiatior, hence VMware snapshots cannot be taken, hence SMVI backups won't be OS-consistent.

Does it make sense now?

Regards,
Radek

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

amritad

Hi Radek,

We have tried to explain app consistency in section 9 of the BPG

http://www.netapp.com/us/library/technical-reports/tr-3737.html

Could you please take a look?

Regards

Amrita

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

radek_kubka

Amrita,

I am not talking about application consistency here. Well, not mainly about this.

Again, there are two choices when using SMVI for virtual OS snapshots:

1) Without VMware snapshots, so OS images are / can be crash-consistent

2) With VMware snapshots, so OS images are in consistent state when NetApp snapshot is taken.

My point is:

If some of your VMs require at the same time SnapManager protection (say for Exchange), then you simply have to use MS iSCSI software initiator, unless you are on SnapDrive 6.2. And MS iSCSI software initiator (as per quoted VMware KB article) is a showstopper for VMware snaps so option 2) is not available.

Of course one may argue: "why bother with OS consistency? it is like a sudden pull of the plug from a server, so checkdisk is the worst what may happen 9 out of 10 times". But this is an entire topic for another discussion

Regards,
Radek

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

watan

SDW 6.1 had support for FCP HBA in ESX server and MS iSCSI initiator in GuestOS for its operations. Now with SDW 6.2 we support ESX iSCSI and iSCSI HBA initiators in SnapDrive. SnapDrive will perform LUN provisioning and snapshot management operations using ESX iSCSI and iSCSI HBA initiators.  The kb from VMware does point out an issue with the configuration you pointed out but there are workarounds in place which hopefully will resolve any issues.

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

marcconeley

Hi everyone.

Some really interesting points made in this thread, but  you've also raised a couple more questions regarding my setup and future use:

1) How do I establish ESX iSCSI LUNs within my VMs? I thought I had to use Microsoft iSCSI initiator LUNs for SnapManager, which may have been true before, but apparently now is not necessary? And until now I have only used MS initiator LUNs.

2) Should I consider converting all my VMs with MS iSCSI LUNs to ESX initiator LUNs and is there any downside to this? (I have a bunch of Win 2003/8 servers running Exchange/SharePoint/SQL etc - all using SnapManager with MS iSCSI LUNs and all currently excluded from SMVI "VM snaps" because it's not supported (see original post).

3) I am still experiencing problems with SMVI creating "VM snaps" of VMs which are running any form of SQL server. Even if the VMs don't have iSCSI initiator LUNs. The event logs of the servers give a VolSnap error:

"The flush and hold writes operation on volume C: timed out while waiting for a release writes command."

I have noticed this is happening only on servers which have an MSQL or MSQL Express engine running (unfortunately, quite a lot of my servers run apps that require a local MSQL Express engine i.e. BackupExec). SnapManager is not installed on these servers.

It was mentioned that VolSnap and VSS conflict and that one or the other should be disabled? Should I do this for these servers and if so how??

Lots of useful info here guys, but i am still a little confused as to how I implement a solution to perform clean, VM snaps of all my servers.

Marc

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

watan

Hi Marc,


1)

Steps to configure the ESX SW iSCSI Software Initiator using VI-Client

    * Enable ESX iSCSI SW initiator in ESX server.
    * Check License under Licensed Features of Configuration Tab.
    * Add the VMKernel Port under the Networking configuration option. Enable      VMotion.
    * Under Security Profile select the port for the Software iSCSI client.
    * Select the ESX SW iSCSI initiator and click the "Enabled" checkbox.

Add ESX SW iSCSI targets to discovery list using VI-Client.

    * Navigate to the Properties of the iSCSI adapter and click on the Dynamic Discovery      tab.
    * Choose to Add new Target.
    * Enter the IP address & port of the target server and click OK.

2) If you're not having any specific problems with using MS iSCSI then I personally don't see any reason to change it.   If its not broken...

Also refer to the following docs as they may have more on performance or general best practices.

The performance best practices :

For ESX 3.5: http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_iscsi_san_cfg.pdf

For ESX 4 : http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf


NetApp and VMware VI3 best practice Guide :

http://media.netapp.com/documents/tr-3428.pd

View solution in original post

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

radnorsan

Radek -

I'm trying to understand this, as I am hopefully going to be implementing a very similar environment.  99% of my VMs are all on our fas2020 on NFS.  All NFS VMs are backed up using SMVI 2.0.  Exchange is the last VM in a lun (on a different filer) connected to ESXi via iSCSI.  I am going to be moving the Exchange VM to the NFS volume soon.  At this time I'll also be changing backup methods to use SM for Exchange.

I'm fully aware about the issues using the MS iSCSI with SMVI.  If I understand what I've read, I should be able to install Snapdrive 6.2 on Exchange to connect the luns to where I will migrate the exchange edb's and logs to, while the boot drive lives in the NFS volume.  Then SMVI will backup the server OS and SM for Ex will backup Exchange?  Is this correct?  Thanks.

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

marcconeley

Sounds good to me!

This is currently how I am doing it too -  my virtual Exchage server is located on an NFS volume (the C:\ if you like), and then I have 2 MS iSCSI LUNs (D:\ and E:\) for my databases, and snapinfo/logs.

I use SMVI to backup the virtual machine (which in effect is the C:\ located on the NFS) and I use SnapManager for Exchange to backup the databases and logs on the iSCSI LUNs.

Together, I have a clean backup of the both the server & the databases.

As you pointed out though, be careful about the performing "vmware consistent snapshots" with SMVI on this Exchange server - I am not sure if this is supported yet.

//Marc

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

radnorsan

Thanks for the help.  If I may ask a bit more?  I am testing this with both the MS iscsi and ESX iscsi initiator and regardless how I do it, SMVI give me a report of "completed with exceptions".  The exception being it couldn't snapshot the disk connected with Snapdrive unless I turn off the VMware snapshots in SMVI (as I expected).

If I may ask, how exactly is your backup strategy laid out?  Currently, all my VMs are in a single NFS volume and SMVI is backing up the entire volume.  This is nice because when I create more VMs I don't have to add them to the backup job.  They get backed up automatically because they are in the volume.

Call me anal, but I don't like SMVI giving me a "completed with exceptions" every time.  I want it to say "successful".  So I could uncheck the VMware snapshot, but I don't want to do that for VMs I don't have to.  That being said, I could put the Exchange VM in its own volume, and have a seperate backup job for the "Exchange" NFS volume (with the vmware snapshot unchecked) and let snapmanager for Exchange handle Exchange itself.  How do you handle it and if you could, how important is having a VMware consistent snapshot for recovery?  Thanks

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

marcconeley

No probs.

This was my original problem (see above) - I was trying to perform VM consistent snapshots of all my servers on the NFS volume - but the ones with iSCSI LUNs would freeze, or the snapshots would fail etc. So to answer you questions on how I do this now:

Thanks for the help.  If I may ask  a bit more?  I am testing this with both the MS iscsi and ESX iscsi  initiator and regardless how I do it, SMVI give me a report of  "completed with exceptions".  The exception being it couldn't snapshot  the disk connected with Snapdrive unless I turn off the VMware snapshots  in SMVI (as I expected).

If I may ask, how exactly is your backup strategy  laid out?  Currently, all my VMs are in a single NFS volume and SMVI is  backing up the entire volume.  This is nice because when I create more  VMs I don't have to add them to the backup job.  They get backed up  automatically because they are in the volume.

I tried a number of different ways to manage this because I also wanted to do VM consistent snapshots. However, creating individual backup jobs (1 consistent snapshot schedule for all VMs without iSCSI LUNs, and 1 for all VMs with iSCSI LUNs i.e. my Exchange & SQL servers) was too much maintenance. I like backing up the entire NFS vol like yourself - and therefore all VMs at the same time - because I don't have to worry about my admins creating new VMs and forgetting to include them in the SMVI schedules.

Call me anal, but I don't like  SMVI giving me a "completed with exceptions" every time.  I want it to  say "successful".  So I could uncheck the VMware snapshot, but I don't  want to do that for VMs I don't have to.  That being said, I could put  the Exchange VM in its own volume, and have a seperate backup job for  the "Exchange" NFS volume (with the vmware snapshot unchecked) and let  snapmanager for Exchange handle Exchange itself.  How do you handle it  and if you could, how important is having a VMware consistent snapshot  for recovery?  Thanks

Yes that would work and its something I've considered (to seperate my iSCSI LUN servers to a seperate NFS volume). 

But rather than trying to seperate my Exchange & SQL servers though I went for the easy option of leaving everything on the same NFS vol, and un-checking the "VM consistent" checkbox in SMVI. This means my server backups are "crash consistent" only and not "VM consistent". Now, how much of an issue this really is is open to debate - and I have never got a full answer. I believe its the same as a dirty shutdown of a server - but I've never had any problems restoring from them. The only problem I see when restoring from these backups is that you see the standard Windows Server error message stating that the last shutdown was unexpected (when your restored VM starts up). In practice though they worked fine.


Therefore I currently accept that all my VM backups are crash consistent only, but this doesn't worry me as all my servers with iSCSI LUNS (SQL & Exchange) are being backed-up with SnapManager seperately. I think this is fine as the databases and logs are in a completely consistent backup state. Obviously what I would never do is perform crash-consistent backups on servers which have their data on local VM disks & then expect to be able to restore the server & it's data. These types of servers need to have their data backed-up via an alternative method (disk dumps, backup agents etc). I never use SMVI as a means for a "data backup" - only as a system backup. I guess if the SMVI snapshots were "VM consistent", therefore application aware, this would be less of a problem?

What I found though was that checking that "VM consistent" checkbox in SMVI caused snapshot failures, VSS error messages on my servers with iSCSI LUNS, and I also saw that many of my VMs would have snapshots left-over that SMVI hadn't deleted - you can see this in the vCenter Console. I believe this is because when this option is used, it's the same as creating a snapshot in VMware and ticking the "quiesce virtual machine" option - in fact, if vCenter console is open you can see it queue all the commands to snapshot the servers when the SMVI schedule runs!

Anyway, the final reason why I stick with crash consistent snapshots was speed and performance. If you have a FAS2020 like me, then telling all 50 of your VMs to perform quiesced snapshots is quite intensive, and takes a fair while - whereas crash consistent snapshots happen on the volume level and are performed by the filer - so they're pretty instantaneous.

Well, that was my reasoning & thinking behind setting up our backup plan. Whether its right or wrong I am not so sure?!!?

I would be interested to know how it goes if you stick with using the "VM consistent" option though; and if you have to switch to using the ESX initiator or stick with the MS one to make it work error free.

I hope this was somewhat helpful!

M



Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

radnorsan

Hello again.  And thanks for such a detailed response.  Sorry it took me so long to respond, I have been doing a number of tests and have even called NetApp support.  The word I got from NetApp was that there is no way around the iSCSI issue with SMVI.  The tech I spoke with kinda put the blame on VMware, but NetApp has a kb article for the same thing.

Anyway, he offered a couple suggestions.  He said I could move the databases out to FC (which I can't do, because we don't have it), or I could try to use OSSV (Open Systems Snapvault), but this looks like it's a seperate product with an additional cost.  Plus, while OSSV still might work, it doesn't look like this is what is is actually meant for.

What I'm leaning towards right now is this:

     Since Exchange is my only server where I will need to do this for, I don't mind using a one-off backup scheme.  I am planning to put Exchange in it's own NFS volume.  I will create a second backup job, that runs every 6 hrs like my current SMVI job, with the VMware snapshots turned off.  This will at least get me a crash consistent backup of Exchange.  Then my plan is to schedule two tasks in vCenter on a Sat or Sun, pick one...  One to power off the Exchange VM and one to power it on (I can afford some downtime on the weekends, around 1:00 AM or so).  So, I will schedule Exchange to shut down at 1:00 AM, then schedule a third SMVI job to start while the VM is shut down, around 1:15 AM or so.  Then schedule the VM to power back up around 1:30.  This will at least allow me to get a good, consistent, backup of Exchange on a weekly basis if my crash-consistent backups have issues.

I know it's a bit of work to setup but I just don't trust the crash-consistent backups when it comes to Exchange.  The end users here are... well ... end users and Exchange is their file storage, email, and lifeblood (I know it's bad, don;t get me started).  Any thought, suggestions, questions, etc.?

Thanks again for your assistance.

Re: SMVI - backing up VMs with iSCSI LUNS and SnapManager

marcconeley

Hi,


Great - it would be nice if you could post something about how it goes. Same if you eventually get the consistent snapshots working on VMs with iSCSI LUNs.

It's strange, because I am sure I read somewhere that the solution to this problem was to use the ESX iSCSI initiator and the latest version of SnapDrive. But you say that the NetApp support actually stated that its a limitation?

Out of interest, do you mind telling me if you're using the MS iSCSI initiator or the ESX one and if you've found any benefits (ease of admin?) in using one or the other?

Your plan to get a clean backup of your Exchange looks fine - if a little extreme! Unfortunately I couldn't do it this way as we have too many SQL/Exchange servers with iSCSI LUNs. Shutting them down to perform a consistent backup isn't possible. One thing you could try instead of shutting down the server, is to simply shutdown all the Exchange services before making the snapshot & then starting them again after - would this work?

I think for the time being I am going to stick with the non VM consistent snapshots - doing a dirty backup of the server is still preferable to having no backup of it. And before we virtualised our Exchange, we were only backing up the dbs with a BackupExec agent anyway - so we have improved our DR capabilities with VMware & NetApp - just a shame that we can't yet use the full SMVI functionality across all our servers (including the ones with iSCSI LUNs).

One thing I am not yet sure about though is this: if you tick this "VM consistent snapshot" in SMVI, then this is the same as ticking "quiesce virtual machine" in the vCenter console? And is ticking "Quiesce virtual machine" in vCenter supported on Exchange/SQL servers? I am not sure...I can't find any definitive answer.

Let me know how it goes & good luck!

Earn Rewards for Your Review!
GPI Review Banner
All Community Forums
Public