ONTAP Hardware

Hot Add Fabric Stretched Mtero Cluster - alter cabling

chrisstyles
11,619 Views

Hi, I am looking for a bit of advice / re assurance 🙂

 

We had an engineer add two new shelves to a fabric stretched metro cluter. Added in fine but he messed up the cabling.

 

Basically he added the new shelves (8 & 9) to the open sas ports. This is correct for the left hand side whch the cabling starts at shelf 0 and the last port is open at shelf 9 now.

 

For the right hand side he connected into the open port on shelf 0 to shelf 9, 9 to 8 and left the open port on shelf 8. (don't ask). He did this on both sides of the metro cluster.

 

So we need to recable the shleves by removing the link (shelf 7 to filer) and the link (shelf 0 to shelf 9) and cabling shelf 9 to the filer and shelf 8 to shelf 7.

 

Question is... if these shelves are now in use and I am chaning the cabling paths as seen by the ATTO bridges, whill I cause any outages? Obviously I will need to do this lve.

 

We intend to disable failover during this process but will loose access to the shelves through the right hand path.

 

Any thoughts or advise greatly appreciated.

1 ACCEPTED SOLUTION

aborzenkov
11,567 Views

Disks on ATTO bridge are named <slot><port>.<loopID>L<LUN>; the <LUN> part enumerates disks starting with the shelf to which ATTO bridge is connected in physical order of shelf attachement (for each side A and B). If you change shelf order in a stack, disks get different LUN numbers. It is not a problem for offline change; but for online - who knows. I appreciate if you could post support response.

View solution in original post

11 REPLIES 11

aborzenkov
11,522 Views

Please show current and intended cabling. For current you can just paste "sasadmin expander_map" output from both filers.

chrisstyles
11,513 Views

Hi,

 

The Cabling goes through an Atto brdge so sasadmin will not show anything.. I have indluded a screen shot of the current cabling.. Thanks

chrisstyles
11,512 Views

And the intended...

aborzenkov
11,493 Views
I would avoid doing it online if possible. It will change disk names, and it was reported to cause issues.

If you absolutely must do it online, open case with netapp and try to assure support and get step by step action plan. It could work if you disable paths through lower bridge and let filer "forget" about them completely; but I'd double check with netapp.

chrisstyles
11,474 Views

OK, the quote, it will change disk names worries me a touch. The disks are named following the fibreswitch name which won't change and the disk serial.

 

I'll open a support case as you suggest and see how it goes. We are going to do one side out of hours so any impact should be minimised.

 

Cheers

aborzenkov
11,568 Views

Disks on ATTO bridge are named <slot><port>.<loopID>L<LUN>; the <LUN> part enumerates disks starting with the shelf to which ATTO bridge is connected in physical order of shelf attachement (for each side A and B). If you change shelf order in a stack, disks get different LUN numbers. It is not a problem for offline change; but for online - who knows. I appreciate if you could post support response.

chrisstyles
11,470 Views

I have created a case and will update you as to how it goes. I really appreciate your input.

 

If the disks have two paths and I am only altering the order of one path... They will get a different LUN number on one path only.. This will get interesting.

 

 data      S1FibreSwitch01:5.126L217       1a    18  8   FC:A   0   SAS 10000 560000/1146880000 572325/1172123568
      data      S1FibreSwitch01:5.126L243       1a    19  8   FC:A   0   SAS 10000 560000/1146880000 572325/1172123568
      data      S1FibreSwitch01:5.126L204       1a    17  21  FC:A   0   SAS 10000 560000/1146880000 560208/1147307696

 

When you say offline, how do you mean? In a Cluster failover?

aborzenkov
11,466 Views

They will get a different LUN number on one path only.. This will get interesting.

They already have different LUN numbers on different paths, at least if cabled according to guidelines. This is common source of confusion in MetroCluster configurations.


When you say offline, how do you mean? In a Cluster failover?

No, takeover does not help here, both controllers always see both paths to shelves. I meant really stopping both nodes.

chrisstyles
11,461 Views

OK, well a halt on both nodes is not an option under any circumstances. So it either doesn't get done or we go for it and risk loosing a stack on one side.

 

I'll post back supports answer. It really does go to show it needs to be done correctly first time round.

chrisstyles
9,970 Views
Netapp support advice.

Power off the atto bridge. Recabled and power the bridge back on to rescan.

As long as the A path is not broken during the work then the filer will use this path for communication.

The filer will cope with the new disk names once they are presented back from the atto.

I'll update what happens in the real world post recabling.

chrisstyles
9,961 Views

Ok, Process followed was this.

 

Send an Autosupport prior to rewire

cf disable

Power Atto bridge off

Check the link light was down for the atto shelf link.

Unwire new shelves

ReWire new shelves

Check Stack uplink lit

Power on Atto

Check Atto link from shelf lit

Wait until system confirm it is back to Multipath HA

cf enable.

 

Simples 🙂

 

I appreciate the help and caution I got on this from the communities and will award as such.

 

Public