Path redundancy to storage device naa.60a98000572d44726d4a52554b735574 degraded. Path vmhba2:C0:T0:L...

dirkslechten · ‎2010-09-07

Hello,

We are working with an IBM6040N connected with 2 bladecenters H.

On a regular bases we receive alerts from our Vcenter :

I have checked the VMkernel logs from the ESX host and the alerts is during for 1 à 2 seconds. On the manage paths in the VCenter all paths are active.

The paths are configured as round robin.

My guess is that something is causing a little stress so there is a small timeout.

Does anybody what could cause this?

Subject: [VMware vCenter - Alarm Cannot connect to storage] Path redundancy to storage device naa.60a98000572d44726d4a52554b735574 degraded. Path vmhba2:C0:T0:L30 is down. 3 remaining active paths. Affected datastores: "VMFS_IBM6040N1A".
To:

Target: ivm2.his-izz.lan
Stateless event alarm

Alarm Definition:
([Event alarm expression: Lost Storage Connectivity] OR [Event alarm expression: Lost Storage Path Redundancy] OR [Event alarm expression: Degraded Storage Path Redundancy])

Event details:
Path redundancy to storage device naa.60a98000572d44726d4a52554b735574 degraded. Path vmhba2:C0:T0:L30 is down. 3 remaining active paths. Affected datastores: "VMFS_IBM6040N1A".

Lun affected:

Datastore:
naa.60a98000572d44726d4a52554b735574 vmhba2:C0:T0:L30

LUN:

VMFS_IBM6040N1A

Subject: [VMware vCenter - Alarm Cannot connect to storage] Path redundancy to storage device naa.60a98000572d44726d4a52554b735574 degraded. Path vmhba2:C0:T0:L30 is down. 3 remaining active paths. Affected datastores: "VMFS_IBM6040N1A".
To:

Target: ivm1.his-izz.lan
Stateless event alarm

Alarm Definition:
([Event alarm expression: Lost Storage Connectivity] OR [Event alarm expression: Lost Storage Path Redundancy] OR [Event alarm expression: Degraded Storage Path Redundancy])

Event details:
Path redundancy to storage device naa.60a98000572d44726d4a52554b735574 degraded. Path vmhba2:C0:T0:L30 is down. 3 remaining active paths. Affected datastores: "VMFS_IBM6040N1A".Lun affected: vmhba2:C0:T0:L30Datastore: naa.60a98000572d44726d4a52554b735574

LUN:

VMFS_IBM6040N1A

Darkstar · ‎2010-09-10

Is this with ESX 4.1? I have seen similar problems there with paths being lost. I think it's a bug in ESX 4.1 that will be fixed in the next release

-Michael

dirkslechten · ‎2010-09-10

No, it's with esx 4.0

basvanberkel · ‎2010-09-10

Do u use ALUA?

igroups show -v

what is esx output from:

# esxcli nmp device

Are your fibreswitches firmware levels up-to-date?
Do u use Virtual Storage Console?

jerome_barrelet · ‎2011-06-30

Hello,

We experience the same problem with ESX 4.1

We use Virtual Storage Console

We use Fiber Channel with ALUA enabled

and yes fibreswitches firmware are up to date.

Somebody have any information about a Bug in ESX 4.1 as Michael say ?

here is the message on vcenter :

Lost connectivity to storage device

naa.60a98000572d50616534646a36745572. Path

vmhba0:C0:T1:L3 is down. Affected datastores:

Unknown.

error

6/30/2011 8:05:37 PM

I did a esxcli nmp device :

naa.60a98000572d50616534645962534139
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d50616534645962534139)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T5:L6, vmhba1:C0:T4:L6, vmhba1:C0:T5:L6

naa.600508b1001030383543314644301000
    Device Display Name: HP Serial Attached SCSI Disk (naa.600508b1001030383543314644301000)
    Storage Array Type: VMW_SATP_LOCAL
    Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
    Path Selection Policy: VMW_PSP_FIXED
    Path Selection Policy Device Config: {preferred=vmhba2:C0:T0:L0;current=vmhba2:C0:T0:L0}
    Working Paths: vmhba2:C0:T0:L0

naa.60a98000572d506165342f2f4a445748
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165342f2f4a445748)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T5:L33, vmhba0:C0:T5:L33, vmhba1:C0:T4:L33

naa.60a98000572d5062684a626e6a744179
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a626e6a744179)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG                              _state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=5: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L15, vmhba1:C0:T0:L15, vmhba0:C0:T0:L15

naa.60a98000572d506165345a742f525a6e
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165345a742f525a6e)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T4:L0, vmhba0:C0:T5:L0, vmhba1:C0:T5:L0

naa.60a98000572d5062684a6459634d4931
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a6459634d4931)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L7, vmhba0:C0:T0:L7, vmhba1:C0:T0:L7

naa.60a98000572d5062684a5a6a5a7a356c
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a5a6a5a7a356c)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L1, vmhba0:C0:T0:L1, vmhba1:C0:T0:L1

naa.60a98000572d50616534617079723561
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d50616534617079723561)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG                              _state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T4:L12, vmhba1:C0:T5:L12, vmhba0:C0:T5:L12

naa.60a98000572d5062684a61707a597a71
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a61707a597a71)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG                              _state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=5: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L13, vmhba1:C0:T0:L13, vmhba0:C0:T0:L13

naa.60a98000572d506165342f2f4a2d756d
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165342f2f4a2d756d)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T5:L31, vmhba0:C0:T5:L31, vmhba1:C0:T4:L31

naa.60a98000572d506165342f2f4a48444a
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165342f2f4a48444a)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T5:L34, vmhba0:C0:T5:L34, vmhba1:C0:T4:L34

naa.60a98000572d506165342f2f4a4b4469
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165342f2f4a4b4469)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T5:L35, vmhba0:C0:T5:L35, vmhba1:C0:T4:L35

naa.60a98000572d5062684a61707a556f38
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a61707a556f38)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L5, vmhba0:C0:T0:L5, vmhba1:C0:T0:L5

naa.60a98000572d506165346468714b4c70
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165346468714b4c70)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=none}
    Working Paths:

naa.60a98000572d506165346170794e4745
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165346170794e4745)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T4:L4, vmhba0:C0:T5:L4, vmhba1:C0:T5:L4

naa.60a98000572d506165342f2f4a354266
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165342f2f4a354266)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T5:L30, vmhba0:C0:T5:L30, vmhba1:C0:T4:L30

naa.60a98000572d50616534626e6a6c7952
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d50616534626e6a6c7952)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG                              _state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T4:L14, vmhba1:C0:T5:L14, vmhba0:C0:T5:L14

naa.60a98000572d5061653464656b436139
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5061653464656b436139)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=none}
    Working Paths:

naa.60a98000572d5062684a5a6a2f782d66
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a5a6a2f782d66)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG                              _state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L11, vmhba1:C0:T0:L11, vmhba0:C0:T0:L11

naa.60a98000572d5062684a5a6a2f2d3457
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d5062684a5a6a2f2d3457)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba0:C0:T1:L3, vmhba0:C0:T0:L3, vmhba1:C0:T0:L3

naa.60a98000572d506165345a742f667941
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165345a742f667941)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG                              _state=ANO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=5: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T4:L10, vmhba1:C0:T5:L10, vmhba0:C0:T5:L10

naa.60a98000572d506165345a742f595663
    Device Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d506165345a742f595663)
    Storage Array Type: VMW_SATP_ALUA
    Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TP                              G_state=AO}}
    Path Selection Policy: VMW_PSP_RR
    Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
    Working Paths: vmhba1:C0:T4:L2, vmhba0:C0:T5:L2, vmhba1:C0:T5:L2

Thanks in advance

Jerome

dirkslechten · ‎2011-06-30

Actually the past was not down, when you check in the vcenter on the affected datastore, all links where up.

Also on the filer, all paths where still active.

As I recall it, it was a time-out issue.

jerome_barrelet · ‎2011-07-04

How did you solve the time-out issue, did you change any parameters in vCenter ?

jerome

ccastaneda · ‎2011-08-05

I'm currently have the same issue and haven't been able to pinpoint this to somekind of bug. I am using ESX 4.1 (No Update anything). I believe this started happening when I enabled ALUA on the igroups. I have this problem in 2 of my 3 sites:

DC1

NetApp FAS3040

ALUA not enabled on igroups

No VSC

Fibre Channel

NO PROBLEM with error messages

DC2

NetApp FAS3140

ALUA enabled on igroups

No VSC

Fibre Channel

PROBLEM with error messages

DC3

NetApp FAS3240

ALUA enabled on igroups

VSC 2.1

Fibre Channel

PROBLEM with error messages

Has anyone been able to find a solution?

Thanks,

jerome_barrelet · ‎2011-08-23

Hello,

We finaly found what append in our vmware infra when we have these kind of messages :

Lost connectivity to storage device

naa.60a98000572d50616534656d6743396f. Path

vmhba1:C0:T0:L3 is down. Affected datastores:

Unknown.

error

It comes from SQL server VM with RDM, when the snapmanager do the DB verification it mount the Cloned lun RDM in the VM, and the messages that you see upper comes when the verification is done and the cloned LUN are removed from the VM.

In my opinion we can safely ignore the messages.

Jerome

NIVEDASRI · ‎2012-06-11

Jerome, You are exactly correct!

It was due to the default Alarm "Cannot Connect to Storage" which triggers an email/notification for any RDM / LUN removal from vCenter/ESXi.

If we want this to fixed, we just need to finetune the vCenter alarm with Advanced settings/parameters to filter False Alarms.

Hope this Helps

Narayan Jayaraman