Hi Everyone,
I just thought I would share the results of my hard work in hopes it will make others lives easier.
I've been working on creating a VMWare SRM 5.1 with NetApp demo environment for over four weeks now. Most of my time has been spent learning SRM and some of the basics of VMware clustering and NFS mounting etc. I finally got to the point where I had:
- 3x VMWare ESXi 5.1 hosts installed
- 2x VCenter Servers 5.1 created with NetApp VSC Plugin Installed
- ESXi Hosts clustered with VMotion working through VCenter
- NFS volumes created and NFS export permissions configured on the NetApp(s)
- NFS Exports mounted as ESXi Datastores
- Win 2008 DC, Exchange 2007 and OCS 2007 VMs installed with 2 Win7 client VMs to prove it was a fully functional demo environment
- NetApp's NFS Volumes SnapMirrored
- SRM installed (ODBC connections to SQL, etc.)
- SRM configured
Moment of truth..
I ran the recovery test plan and it failed with this Error - Failed to create snapshots of replica devices. Failed to create snapshot of replica device /vol/NetApp_Datastore1. SRA command 'testFailoverStart' failed for device '/vol/NetApp_Datastore1'. Unable to export the NAS device Ensure that the correct export rules are specified in the ontap_config.txt file.
(See the SRM Failures.doc in the attachment) I was at this point on Monday. I scoured the internet for this error and found very limited information. I also obsessed over the ontap_config.txt file on the VCenter/SRM server to no avail. What I did find is when configuring NFS every Volume/Export must be explicitly configured with r/w & root privileges. RO must be removed along with Anonymous access. The export should look like the "Permissions.jpg" in the attachment. From the command line the permissions should look like this:
ONTAP-SRM-2> exportfs
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
/vol/SRM_Placeholder -sec=sys,rw=10.18.202.31,root=10.18.202.31
/vol/NetApp_Datastore1 -sec=sys,rw=10.18.201.31,root=10.18.201.31
/vol/NetApp_Datastore2 -sec=sys,rw=10.18.202.31,root=10.18.202.31
That being said. How SRM works when you execute the recovery test plan is it creates a FlexClone of the SnapMirrored volume. When I executed the test plan and it failed I started looking at the NetApp on the recovery site. I noticed it created the FlexClone but it obviously didn't mount it as a datastore and start up the VMs. Funny thing is that when I manually mounted the FlexClone I could start up the VMs.
Thinking that NFS is all about permissions... look what I found on the permissions of the FlexClone when I ran the "exportfs" command on the recovery side NetApp
ONTAP-SRM-2> exportfs
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore1 -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore2 -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
/vol/SRM_Placeholder -sec=sys,rw=10.18.202.31,root=10.18.202.31
/vol/NetApp_Datastore1 -sec=sys,rw=10.18.201.31,root=10.18.201.31
/vol/NetApp_Datastore2 -sec=sys,rw=10.18.202.31,root=10.18.202.31
Look at all of that junk in there.. for anyone who doesn't recognize this.. This is IPv6 and I am pretty sure SRM 5.1 doesn't work with IPv6.
SOLUTION - on the ESXi hosts:
To view whether IPv6 is currently enabled, run the following ESXCLI command:
esxcli system module parameters list -m tcpip3
You will see, ipv6 property is set to 1 which means it is enabled.
To disable IPv6, you just need to set the property to 0, run the following ESXCLI command:
esxcli system module parameters set -m tcpip3 -p ipv6=0
We can now reconfirm by re-running our list operation to ensure the changes were made successfully. All that is left is to perform a system reboot, you can either type in "reboot" or use the new ESXCLI 5.1 command:
esxcli system shutdown reboot -d 60 -r "making IPv6 config changes"
After the ESXi host reboots see the FlexClone permissions:
ONTAP-SRM-2> exportfs
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore1 -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore2 -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
/vol/SRM_Placeholder -sec=sys,rw=10.18.202.31,root=10.18.202.31
/vol/NetApp_Datastore1 -sec=sys,rw=10.18.201.31,root=10.18.201.31
/vol/NetApp_Datastore2 -sec=sys,rw=10.18.202.31,root=10.18.202.31
ONTAP-SRM-2>
Note IPv6 permissions are now gone = Success! SRM Test Recover Plan works!
Enjoy.
