ONTAP Hardware

VMWare SRM 5.1 with NetApp SRA 2.0.1 over NFS - Error: Failed to create snapshots of replica devices; Unable to export the NAS device Ensure that the correct export rules are specified in the ontap_config.txt file

CCIE17603
7,262 Views

Hi Everyone,

I just thought I would share the results of my hard work in hopes it will make others lives easier.

I've been working on creating a VMWare SRM 5.1 with NetApp demo environment for over four weeks now. Most of my time has been spent learning SRM and some of the basics of VMware clustering and NFS mounting etc. I finally got to the point where I had:

- 3x VMWare ESXi 5.1 hosts installed

- 2x VCenter Servers 5.1 created with NetApp VSC Plugin Installed

- ESXi Hosts clustered with VMotion working through VCenter

- NFS volumes created and NFS export permissions configured on the NetApp(s)

- NFS Exports mounted as ESXi Datastores

- Win 2008 DC, Exchange 2007 and OCS 2007 VMs installed with 2 Win7 client VMs to prove it was a fully functional demo environment

- NetApp's NFS Volumes SnapMirrored

- SRM installed (ODBC connections to SQL, etc.)

- SRM configured

Moment of truth..

I ran the recovery test plan and it failed with this Error - Failed to create snapshots of replica devices. Failed to create snapshot of replica device /vol/NetApp_Datastore1. SRA command 'testFailoverStart' failed for device '/vol/NetApp_Datastore1'. Unable to export the NAS device Ensure that the correct export rules are specified in the ontap_config.txt file.

(See the SRM Failures.doc in the attachment) I was at this point on Monday. I scoured the internet for this error and found very limited information. I also obsessed over the ontap_config.txt file on the VCenter/SRM server to no avail. What I did find is when configuring NFS every Volume/Export must be explicitly configured with r/w & root privileges. RO must be removed along with Anonymous access. The export should look like the "Permissions.jpg" in the attachment. From the command line the permissions should look like this:

ONTAP-SRM-2> exportfs
/vol/vol0/home  -sec=sys,rw,nosuid
/vol/vol0       -sec=sys,rw,anon=0,nosuid
/vol/SRM_Placeholder    -sec=sys,rw=10.18.202.31,root=10.18.202.31
/vol/NetApp_Datastore1  -sec=sys,rw=10.18.201.31,root=10.18.201.31
/vol/NetApp_Datastore2  -sec=sys,rw=10.18.202.31,root=10.18.202.31

That being said. How SRM works when you execute the recovery test plan is it creates a FlexClone of the SnapMirrored volume. When I executed the test plan and it failed I started looking at the NetApp on the recovery site. I noticed it created the FlexClone but it obviously didn't mount it as a datastore and start up the VMs. Funny thing is that when I manually mounted the FlexClone I could start up the VMs.

Thinking that NFS is all about permissions... look what I found on the permissions of the FlexClone when I ran the "exportfs" command on the recovery side NetApp

ONTAP-SRM-2> exportfs
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore1  -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore2  -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31:fe80::20c:29ff:fea5:ac48:fe80::250:56ff:fe63:7152:fe80::250:56ff:fe66:28d7:fe80::250:56ff:fe66:fd86
/vol/vol0/home  -sec=sys,rw,nosuid
/vol/vol0       -sec=sys,rw,anon=0,nosuid
/vol/SRM_Placeholder    -sec=sys,rw=10.18.202.31,root=10.18.202.31
/vol/NetApp_Datastore1  -sec=sys,rw=10.18.201.31,root=10.18.201.31
/vol/NetApp_Datastore2  -sec=sys,rw=10.18.202.31,root=10.18.202.31

Look at all of that junk in there.. for anyone who doesn't recognize this.. This is IPv6 and I am pretty sure SRM 5.1 doesn't work with IPv6.

SOLUTION - on the ESXi hosts:

To view whether IPv6 is currently enabled, run the following ESXCLI command:

esxcli system module parameters list -m tcpip3

You will see, ipv6 property is set to 1 which means it is enabled.

To disable IPv6, you just need to set the property to 0, run the following ESXCLI command:

esxcli system module parameters set -m tcpip3 -p ipv6=0

We can now reconfirm by re-running our list operation to ensure the changes were made successfully. All that is left is to perform a system reboot, you can either type in "reboot" or use the new ESXCLI 5.1 command:

esxcli system shutdown reboot -d 60 -r "making IPv6 config changes"

After the ESXi host reboots see the FlexClone permissions:

ONTAP-SRM-2> exportfs
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore1  -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31
/vol/testfailoverClone_nss_v10745371_NetApp_Datastore2  -sec=sys,rw=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31,root=10.18.18.31:10.18.201.31:10.18.202.31:10.18.203.31
/vol/vol0/home  -sec=sys,rw,nosuid
/vol/vol0       -sec=sys,rw,anon=0,nosuid
/vol/SRM_Placeholder    -sec=sys,rw=10.18.202.31,root=10.18.202.31
/vol/NetApp_Datastore1  -sec=sys,rw=10.18.201.31,root=10.18.201.31
/vol/NetApp_Datastore2  -sec=sys,rw=10.18.202.31,root=10.18.202.31
ONTAP-SRM-2>

Note IPv6 permissions are now gone = Success! SRM Test Recover Plan works!

Enjoy.

4 REPLIES 4

aborzenkov
7,262 Views

Interesting. It is not actually crap, but square brackets are missing. It should have been :[fe80::20c:29ff:fea5:ac48]:[fe80::250:56ff:fe63:7152]:[fe80::250:56ff:fe66:28d7]:[fe80::250:56ff:fe66:fd86]

Looks like a bug in SRA somewhere. Disabling IPv6 in ESX is just a workaround, but it needs to be solved properly. Did you open case with NetApp?

CCIE17603
7,262 Views

No I didn't. I wasn't sure if it was a VMWare issue (SRM) or NetApp (SRA). Do you know if IPv6 is even supported by the automation flow of SRM 5.1? None of the documentation I read referred to IPv6, nor any quick internet searches. The only place it referred to IPv6 in the installation and configuration guide for 5.1 is in chapter 6 on page 58 when it refers to the public certificate for vSphere Replication.

However, I know it does support IPv6 when customizing the IPs for the virtual machines themselves. This is explained clearly in the administration guide.

Regards.

carnevja
7,262 Views

This is the only post on the interwebs that mentions this issue.  Thank you so much for the info!

MIKE_ELLIS
7,262 Views

I am getting a similar issue but the reason the fails to create the replica is because NFS is licensed at 1 site but not at the other. Keep in mind the volumes are all FC mounted so this should not be an issue. Here are the errors I and seeing.

from destination controller:

exportfs [Line 2]: NFS not licensed; local volume /vol/fc_site1fas1_esx_srm_datastore_srm_test not exported

exportfs [Line 2]: NFS not licensed; local volume /vol/fc_site1fas1_esx_srm_datastore_srm_test not exported

exportfs [Line 2]: NFS not licensed; local volume /vol/fc_site1fas1_esx_srm_datastore_srm_test not exported

exportfs [Line 2]: NFS not licensed; local volume /vol/fc_site1fas1_esx_srm_datastore_srm_test not exported

from source controller:

Wed Jun 12 17:41:53 EDT [site2fas1:app.log.info:info]: SRM01: NetApp FAS/V-Series Storage Replication Adapter 2.0.1P2: (3) Test-Failover-stop Event: NetApp FAS/V-Series Storage Replication Adapter executed Test-Failover-stop operation successfully from OS major version = 6 ,minor version = 1 ,package = Service Pack 1 and build = 7601  

Wed Jun 12 17:42:26 EDT [site2fas1:app.log.info:info]: SRM01: NetApp FAS/V-Series Storage Replication Adapter 2.0.1P2: (2) Test-Failover-start Event: NetApp FAS/V-Series Storage Replication Adapter executed Test-Failover-start operation with errors from OS major version = 6 ,minor version = 1 ,package = Service Pack 1 and build = 7601  

Wed Jun 12 17:42:26 EDT [site2fas1:callhome.client.app.info:info]: Call home for CLIENT APP INFO Test-Failover-start Event: NetApp FAS/V-Series Storage Replication Adapter 2.0.1P2: (2) on SRM01: NetApp FAS/V-Series Storage Replication Adapter executed Test-Failover-start operation with errors from OS major version = 6 ,minor version = 1 ,package = Service Pack 1 and build = 7601  

Wed Jun 12 17:51:12 EDT [site2fas1:app.log.info:info]: SRM01: NetApp FAS/V-Series Storage Replication Adapter 2.0.1P2: (3) Test-Failover-stop Event: NetApp FAS/V-Series Storage Replication Adapter executed Test-Failover-stop operation successfully from OS major version = 6 ,minor version = 1 ,package = Service Pack 1 and build = 7601  

Public