Volume not on preferred path due to AVT/RDAC failover

TandV · ‎2016-04-29

Hello everyone,

our NetApp E2724 is direct attached to two Cisco C240 M4 Server with ESXi 6.0.0. The volume configuration is one Volume Group with 2 Volumes. For Volume “A” the preferred controller is A and for Volume “B” the preferred controller is B.

Everything is working fine, also the failover but I always get the critical error message “Volume not on preferred path due to AVT/RDAC failover”.

Configuration (see attachment):

2 x VmWare ESXi 6.0.0

NetApp E2724

iSCSI (2 vSwitches) direct attached

Path Selection: Round Robin (also tried “Fixed”)

Storage Array Type: VMW_SATP_ALUA

Does anyone know what could be done to solve this problem? OR is there a workaround?

Makes it sense to disable ALUA? (and how can I disable ALUA?)

Thanks for the assistance,

Regards TandV

MikeSamples · ‎2016-05-10

Thats not a big deal... You just need to go into SANtricity, highligt the volume and right mouse click. One of the options will be to change ownership / preferred path. Change it to A or B whatever the preferred path should be.

Mike

Randy_Lin · ‎2016-07-22

The way you mentioned was done for many times. But it keeps happen almost everyday.

This is the real problem that we are facing.

What shoud we do to that?

Thanks in advance.

ZboinaJ · ‎2016-08-10

Any update? I'm occuring same issue and have no idea what to do

MikeSamples · ‎2016-08-10

If it keeps happening then there is a problem causing it. I would make sure the failover drivers are setup correctly on all system connecting to those ports. It only takes 1 system not configured correctly to cuase a issues with all luns presenting down that path.

Follow this guide: https://library.netapp.com/ecm/ecm_get_file/ECMP1394845

@ZboinaJ wrote:
Any update? I'm occuring same issue and have no idea what to do

Randy_Lin · ‎2016-08-11

Hi, MikeSamples:

Thanks for your help.

My system setup is .... 3 volumes were created. 2 of them are accessed via Controller A and wroks great.

But the 1 of them that accessed via Controller B , called "Netapp-E2724_ESXi6-2", keeps sending that "Volume not on preferred path due to AVT/RDAC failover" messages and switch to Controller A everyday.

Before the E2724 change the path of volume "Netapp-E2724_ESXi6-2", the volume works normally.

What can I do to find out the root cause?

Some system information ...

At ESXi side ...

Result of command "esxcli storage nmp device list"

naa.600a098000a07efb000000dc5717f5f8 (E2724 ESXi6-2)
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098000a07efb000000dc5717f5f8)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}{TPG_id=0,TPG_state=ANO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba2:C0:T4:L1, vmhba3:C0:T4:L1
   Is USB: false

At E2724 side ...

MAPPINGS (SANshare Storage Partitioning - Enabled (1 of 128 used))-------------------

   Volume Name           LUN Controller Accessible by      Volume status DA Enabled Volume Capacity Type
   Access Volume         7    A,B         Default Group      Optimal        NA                           Access
   Access Volume         7    A,B         Host Group VMware Optimal        NA                           Access
   Netapp-E2724_ESXi6-1 0    A           Host Group VMware Optimal        Yes         3,072.000 GB     Standard
   Netapp-E2724_ESXi6-2 1    A           Host Group VMware Optimal        Yes         3,072.000 GB     Standard
   Netapp-E2724_ESXi6-3 2    A           Host Group VMware Optimal        Yes         3,072.000 GB     Standard

   TOPOLOGY DEFINITIONS

      STORAGE ARRAY
         Default type:                       Factory Default

         Host Group:                         VMware
            Data Assurance (DA) capable:     Yes

            Host:                            BE15
               Host type:                    Windows
               Interface type:               Fibre Channel
                  Host port identifier:      50:01:43:80:05:67:49:62
                  Alias:                     BE15_p1
                  Host port identifier:      50:01:43:80:05:67:48:2a
                  Alias:                     BE15_p2
               Data Assurance (DA) capable: Yes
               Large sector size supported: No
            Host:                            ESXi6-3
               Host type:                    VMWare
               Interface type:               Fibre Channel
                  Host port identifier:      50:01:43:80:00:c5:2f:dc
                  Alias:                     ESXi6-3_p1
                  Host port identifier:      50:01:43:80:00:c5:30:38
                  Alias:                     ESXi6-3_p2
               Data Assurance (DA) capable: Yes
               Large sector size supported: No
            Host:                            ESXi6-1
               Host type:                    VMWare
               Interface type:               Fibre Channel
                  Host port identifier:      50:01:43:80:00:c4:6d:cc
                  Alias:                     ESXi6-1_p1
                  Host port identifier:      50:01:43:80:00:c4:6d:68
                  Alias:                     ESXi6-1_p2
               Data Assurance (DA) capable: Yes
               Large sector size supported: No
            Host:                            ESXi6-2
               Host type:                    VMWare
               Interface type:               Fibre Channel
                  Host port identifier:      50:01:43:80:03:ae:a4:9a
                  Alias:                     p1
                  Host port identifier:      50:01:43:80:03:ae:a4:2c
                  Alias:                     p2
               Data Assurance (DA) capable: Yes
               Large sector size supported: No


   NVSRAM HOST TYPE DEFINITIONS

   NOTE: The following indexes are not used: 3 - 4, 11 - 14, 16, 19 - 21, 28 - 31

   HOST TYPE                            ALUA/AVT STATUS
   AIX MPIO                             Disabled         9
   AVT_4M                               Enabled          5
   Data ONTAP (ALUA)                    Enabled          26
   Factory Default                      Disabled         0 (Default)
   HP-UX                                Enabled          15
   Linux (ATTO)                         Enabled          24
   Linux (DM-MP)                        Enabled          7
   Linux (MPP/RDAC)                     Disabled         6
   Linux (Pathmanager)                  Enabled          25
   Linux (Symantec Storage Foundation) Enabled          27
   Mac OS                               Enabled          22
   SVC                                  Enabled          18
   Solaris (v11 or later)               Enabled          17
   Solaris (version 10 or earlier)      Disabled         2
   VMWare                               Enabled          10
   Windows                              Enabled          1
   Windows (ATTO)                       Enabled          23
   Windows Clustered                    Enabled          8

Nacino_Alexander · ‎2016-08-30

We are receiving the same error on our end. Were you able to resolve this?

Randy_Lin · ‎2016-08-30

You should check the file major-event-log.txt extracted from auto-support-file.7z.

In my case, it was caused by IO shipping. We found the backup server only have a single path connected to Controller A, the backup flow causes the IO shipping , and that makes the volumes owned by Controller B be changed to controller A. Eventually, we set preferred path to Controller A on all volumes.

You can find more information in KB Doc ID 1015723 . it describes how it happens and how to resolve it.

======= my major-event-log.txt =======

Date/Time: 2016/8/20 下午 11:41:58
Sequence number: 2429
Event type: 4011
Event category: Error
Priority: Critical
Event needs attention: true
Event send alert: true
Event visibility: true
Description: Volume not on preferred path due to AVT/RDAC failover
Event specific codes: 0/0/0
Component type: Controller
Component location: Tray 99, Slot A
Logged by: Controller in slot A

Date/Time: 2016/8/20 下午 11:36:58
Sequence number: 2428
Event type: 2044
Event category: Internal
Priority: Informational
Event needs attention: false
Event send alert: false
Event visibility: true
Description: IO shipping implicit volume transfer
Event specific codes: 0/0/0
Component type: Volume
Component location: Volume E2724_VM-2
Logged by: Controller in slot B

Nacino_Alexander · ‎2016-08-30

I can't see the KB ID based on the link you provided. Looks like we receive the same alert.

Randy_Lin · ‎2016-08-31

How about searching KB id 1015723 in https://kb.netapp.com/support/index?page=home