topic Re: All Paths Down (APD) on controller fail over in VMware Solutions Discussions

All Paths Down (APD) on controller fail over

davidrnexon — Tue, 01 Nov 2016 22:59:30 GMT

Hi,

We have noticed within VMware (version is 5.5. update 3 with NFS datastores) that during a controller failover (i.e. testing or Data Ontap upgrade) that a few datastores experience the All Paths Down (APD) event for about 10 seconds.

NFS.MaxQueueDepth is set at 64 currently.

The APD events only occur on controller failover and not during normal operations.

Has anyone experienced this on controller fail overs, and has anyone had success with eliminating APD events by dropping the NFS.MaxQueueDepth to 32 or less ?

Thank you

Re: All Paths Down (APD) on controller fail over

SeanHatfield — Wed, 02 Nov 2016 01:18:52 GMT

Which version of ONTAP?

7 or C?

and which model?

Have you applied the settings from the VSC and rebooted the hosts?

How are the switchports configured?

Sounds like you're reading vmware kb 2016122.

Re: All Paths Down (APD) on controller fail over

davidrnexon — Wed, 02 Nov 2016 03:07:14 GMT

Hey Sean - Clustered Ontap 8.3.2P5 Model AFF 8060, VSC setting have been applied. Yes VSC settings have been applied and hosts rebooted.

Switches are 5k's VPC port channel for NFS ifgrp on the AFF's. 1 port from each netapp port channel to 5k switch 1 and 2 - MTU 9000 throughout.

I have read that article but that article seems to of been written if you are experiencing random APD's through a normal day.

I have a support ticket open with VMware - who have suggested the drop in MaxQueueDepth to 32 based on what the storage vendor says.

I also have a ticket with Netapp which has not gotten very far

Re: All Paths Down (APD) on controller fail over

SeanHatfield — Wed, 02 Nov 2016 20:30:27 GMT

That article does seem like a red herring in your situation.

Does it also happen during a lif migrate across nodes? or is it just at failover?

It sounds like you've got support of the vendors on the ends engaged, just don't overlook the network in-between; flow control, portfast/spanning tree port type edge trunk, mtu 9216, etc.

Re: All Paths Down (APD) on controller fail over

davidrnexon — Thu, 03 Nov 2016 00:07:39 GMT

Don't see any APD's during a lif migrate operation

Network components look all good

We have found in the logs that during the failover the NFS service is taking about 5 seconds to startup on the active node. We are going to drop the MaxQueueDepth to 32 on an ESXi host and test failover again.

Re: All Paths Down (APD) on controller fail over

SeanHatfield — Thu, 03 Nov 2016 01:32:41 GMT

Does that SVM have an NFS lif on that node in a non-failover state?

Re: All Paths Down (APD) on controller fail over

xkursion — Mon, 24 Jul 2017 21:32:41 GMT

Experiencing a similar issue here during failover testing. Were you able to resolve? Have tickets open with VMware.

Thx

Re: All Paths Down (APD) on controller fail over

davidrnexon — Mon, 24 Jul 2017 23:33:14 GMT

I also had the same - tickets open with Netapp and VMware

Basically it came down to the failover time between controllers. Sometimes the failover time was quick (in this case we didn't experience any APD) in other times it was slightly slower (depending on how busy the controller was at the time) which led to an APD. Basically there is was no guarantee that the failover was going to occur faster or slower at the time. Really depends or comes down to how busy the controller is. You can see the failover times for individual protocols in ::> event log show -event *nfs* (after failover)

Out of interest what controllers are you running, and what's their utilization like ?

Re: All Paths Down (APD) on controller fail over

xkursion — Tue, 25 Jul 2017 14:12:48 GMT

Running AFF8040 9.1P2

Very low utilization currently. Just got everything stood up, issue presented during some failover testing prior to production use. Typically seeing peaks of <10MBps throughput and <10k IOPS.