Hot Add Fabric Stretched Mtero Cluster - alter cabling

[chrisstyles]

Hi, I am looking for a bit of advice / re assurance 🙂

We had an engineer add two new shelves to a fabric stretched metro cluter. Added in fine but he messed up the cabling.

Basically he added the new shelves (8 & 9) to the open sas ports. This is correct for the left hand side whch the cabling starts at shelf 0 and the last port is open at shelf 9 now.

For the right hand side he connected into the open port on shelf 0 to shelf 9, 9 to 8 and left the open port on shelf 8. (don't ask). He did this on both sides of the metro cluster.

So we need to recable the shleves by removing the link (shelf 7 to filer) and the link (shelf 0 to shelf 9) and cabling shelf 9 to the filer and shelf 8 to shelf 7.

Question is... if these shelves are now in use and I am chaning the cabling paths as seen by the ATTO bridges, whill I cause any outages? Obviously I will need to do this lve.

We intend to disable failover during this process but will loose access to the shelves through the right hand path.

Any thoughts or advise greatly appreciated.

[aborzenkov]

Please show current and intended cabling. For current you can just paste "sasadmin expander_map" output from both filers.

[chrisstyles]

Hi,

The Cabling goes through an Atto brdge so sasadmin will not show anything.. I have indluded a screen shot of the current cabling.. Thanks

[chrisstyles]

And the intended...

[aborzenkov]

I would avoid doing it online if possible. It will change disk names, and it was reported to cause issues.

If you absolutely must do it online, open case with netapp and try to assure support and get step by step action plan. It could work if you disable paths through lower bridge and let filer "forget" about them completely; but I'd double check with netapp.

[chrisstyles]

OK, the quote, it will change disk names worries me a touch. The disks are named following the fibreswitch name which won't change and the disk serial.

I'll open a support case as you suggest and see how it goes. We are going to do one side out of hours so any impact should be minimised.

Cheers

[aborzenkov · Accepted solution]

Disks on ATTO bridge are named <slot><port>.<loopID>L<LUN>; the <LUN> part enumerates disks starting with the shelf to which ATTO bridge is connected in physical order of shelf attachement (for each side A and B). If you change shelf order in a stack, disks get different LUN numbers. It is not a problem for offline change; but for online - who knows. I appreciate if you could post support response.

[chrisstyles]

I have created a case and will update you as to how it goes. I really appreciate your input.

If the disks have two paths and I am only altering the order of one path... They will get a different LUN number on one path only.. This will get interesting.

 data      S1FibreSwitch01:5.126L217       1a    18  8   FC:A   0   SAS 10000 560000/1146880000 572325/1172123568
      data      S1FibreSwitch01:5.126L243       1a    19  8   FC:A   0   SAS 10000 560000/1146880000 572325/1172123568
      data      S1FibreSwitch01:5.126L204       1a    17  21  FC:A   0   SAS 10000 560000/1146880000 560208/1147307696

When you say offline, how do you mean? In a Cluster failover?

[aborzenkov]

---

They will get a different LUN number on one path only.. This will get interesting.

---

They already have different LUN numbers on different paths, at least if cabled according to guidelines. This is common source of confusion in MetroCluster configurations.

---

When you say offline, how do you mean? In a Cluster failover?

---

No, takeover does not help here, both controllers always see both paths to shelves. I meant really stopping both nodes.

[chrisstyles]

OK, well a halt on both nodes is not an option under any circumstances. So it either doesn't get done or we go for it and risk loosing a stack on one side.

I'll post back supports answer. It really does go to show it needs to be done correctly first time round.

[chrisstyles]

Netapp support advice.

Power off the atto bridge. Recabled and power the bridge back on to rescan.

As long as the A path is not broken during the work then the filer will use this path for communication.

The filer will cope with the new disk names once they are presented back from the atto.

I'll update what happens in the real world post recabling.

[chrisstyles]

Ok, Process followed was this.

Send an Autosupport prior to rewire

cf disable

Power Atto bridge off

Check the link light was down for the atto shelf link.

Unwire new shelves

ReWire new shelves

Check Stack uplink lit

Power on Atto

Check Atto link from shelf lit

Wait until system confirm it is back to Multipath HA

cf enable.

Simples 🙂

I appreciate the help and caution I got on this from the communities and will award as such.