Physically relocate nodes and disk shelves of a 4 nodes cluster

[storageguy]

Dear NetApp,

Due to rack requirement for new hardware I need to relocate 2 nodes (FAS8040) and 6 disk shelves to a different rack. I am looking for steps if anyone has done this before. I am thinking of:

1. Disable HA and shutdown 4 nodes

2. Then turn the power off on disk shelves

3.  Then turn off cluster switches.

4. Physically relocate two nodes and 6 disks shelves

6. Make sure cluster/network connections are patched in

7. Start Cluster switches and power-up disks shelves (wait until spinning cycle completes)

8. Power up each node

9. Enable HA

Am I missing anything?

Thanks

[Mjizzini]

If you can schedule a downtime, this action plan is perfectly fine.

[storageguy]

Thanks for the replay. 

My architect said we can shut down one HA pair in the 4 nodes cluster and connected disk shelves while other HA pair and disk shelves running ( I need to relocate one HA pair only), Is it possible?

Thanks

[TMACMD]

You actually do not have to disable HA. 

Shut down the ha pair first but remember to relocate epsilon

 set advanced

 cluster show-> is epsilon on a node being shut down?

 cluster modify -node node-with-epsilon -epsilon false

cluster modify -node node-staying-up -epsilon true

 then just shutdown and inhibit takeover of the two nodes

 halt -node node1 -inhibit-takeover true

halt -node node2 -inhibit-takeover true

(i think in diag mode there may even be a -poweroff option also)

 label all you cables. Verify they are secure when racking back together!

 Might want to boot into maintenance mode on both nodes to verify the aggregates are good before letting ontap boot on its own

[storageguy]

Ok, Thanks. 

My issue is if shutdown entire cluster I will need to power-down 270+ VMs.  two nodes and 11 shelfs are half of the cluster. The disk shelves, I need to power-down are only connected to the nodes that will be relocated. So my questions is it possible to run the cluster with with other  pair and it disk shelves while other two down.

Ex:  Cluster has node a and node b - HA pair with disk shelfs 00-04 and node c and node d HA pair with disk shelfs 30-36. I want to shutdown node c and node d HA pair disk shelfs 30-36 to relocate. Can I just shutdown node c and node d with their disk shelfs while node a and node b running with shelfs 00-04?

Hope my question make sense.

Thanks again

[aborzenkov]

---

@storageguy wrote:

Can I just shutdown node c and node d with their disk shelfs while node a and node b running with shelfs 00-04?

---

Yes and now. To be fully functional cluster needs quorum - more than half of nodes. If you lose exactly half of all nodes, remaining nodes will retain quorum thanks to epsilon. But  when you power on half of the nodes on another location, they will not be usable until quorum is attained, so no access to data on these nodes will be possible. So at the end you gain nothing.

So yes, you can shut down one HA pair. No, you cannot bring this HA pair up on another location separately.

[storageguy]

To be clear, I will not create a new cluster.  After relocation nodes and disk shelves will be in the same cluster as before the relocation. Cluster will be same  as before relocation but nodes c and d will be in different rack with their shelves. 

Thanks 

[TMACMD · Accepted solution]

Here is the bottom line:

Make SURE that epsilon is on a NODE that will continue to stay up.

Halt the two nodes to relocate.

Power off only the gear being relocated

Relocate the gear

Re-cable all the gear. Be sure they are still connected to the cluster switches (they will not operate in any stand alone fashion)

Power on.

If possible, boot the nodes to MAINT mode and verify aggregates are still OK after the move. (then halt)

Boot into ONTAP. THey should join back into the cluster.

[storageguy]

Thanks for the input guys. Now I will go and prepare pre-relocation tasks

-Relocate some volumes to staying up aggregates

- Failover LIFs to the staying up nodes

-Make sure SVMs not hosting root volume on the shutdown nodes aggregates.

That is the life I guess when new NetApp hardware changes dimensions so some standard racks become unsuitable.