Re: Stack Same Path warning... recabling advice.

DUANESTRYDER · ‎2013-02-16

I am getting "Stack Same path" warning when running Config Advisor. This is because of the connectivity of the primary and redundant path on four stack of FC shelves look like this:

1d, 1c

3d, 3c

11b, 11a

4d, 4c

I guess what this means, is if the module (example IOM on port 1 where 1d and 1c) breaks, then I lose the entire stack, am I right? potential loss of data? especially since we are using software base ownership (disk assign)

I plan to recable them, but I want to learn more, but I plan to recable them like these:

1d, 3c

3d, 11a

11b, 4c

4d, 1c

What will be the best approach on recabling them? Total shutdown? anything else need to be done on the OS level?

Thanks,

md

bbjholcomb · ‎2013-02-16

Keep in mind that most cards (including the onboard ports) have one ASIC chip for two ports, you shouldn't use A&B or C&B ports from the same card on one controller.

You might find easier to cable them using same port such on two different cards, for example 1a/3a, 4a/11a partner could be 1b/3b, 4b/11b. You have a lot of choices.

DUANESTRYDER · ‎2013-02-18

hi, could you please elaborate on why it is not advisable to use A&B or C&B?

bbjholcomb · ‎2013-02-18

You are looking for the A&B modules of the disk shelf. I don't understand what you mean by C&B.

For example:

PRIMARY PORT SECONDARY PORT SHELF BAY

-------- ---- --------- ---- ---------

4a.24.0 B 3a.24.0 A 24 0

4a.24.1 B 3a.24.1 A 24 1

4a.24.2 B 3a.24.2 A 24 2

In this example B module is the primary and A is the secondary.

DUANESTRYDER · ‎2013-02-18

Hi,

I was just responding to your statement

"you shouldn't use A&B or C&B ports from the same card on one controller.You might find easier to cable them using same port such on two different cards, for example 1a/3a, 4a/11a partner could be 1b/3b, 4b/11b".

I understand the part where you say that I should use port a on different cards. I thought you were saying that I should not use port 'a' on one card, then port 'c' on another card.

bbjholcomb · ‎2013-02-18

Typo, I meant ports A&B are on one ASIC and ports C&D another ASIC.

DUANESTRYDER · ‎2013-02-18

thanks for clarifying. I heard that advise somewhere also but not so sure the reason behind it.

If I change the redundanth path, will I need to do anything else or will it be auto-detected without interruption.

Thanks,

md

bbjholcomb · ‎2013-02-19

Once you verify MPHA you should be able to move cables without down time. Do it slowly, one cable at a time then make sure everything is working before moving another cable, storage show disk -p before each move.

DUANESTRYDER · ‎2013-02-20

Hi,

in your sample above, B module is the primary. What determines between A and B module as to which becomes the primary and the redundant?

DUANESTRYDER · ‎2013-02-20

I guess I am now confusing the primary and secondary to the cable redundancy. The primary is the module connected to the owner of the disk. is that correct?

bbjholcomb · ‎2013-02-20

Ontap does this automatically. Once you move the fiber ports you will see it change.

DUANESTRYDER · ‎2013-02-20

I found this link. I am not quite sure if the explanation of the FC multipathing for an HA pair is accurate or just confusing me even more.

http://blog.hernanjlarrea.com.ar/index.php/netapp-cabling-and-hardware-basics/

bbjholcomb · ‎2013-02-21

I know people at NetApp struggled with MPHA at first; the concept is difficult to understand at first, once you have worked with it for a while it will make sense. The link you have is about the best explanation you will find; the diagrams are very good at the end of the article. The old shelves had IN/OUT on the modules; they changed at some point to arrows.

If you send me you layout I might be able to help more. Let me know the type of modules you are using in each stack. We have a lot of equipment onsite, I can send pictures if that helps.

You can always get NetApp PS services to help you with this, I'm sure they will charge for this.

DUANESTRYDER · ‎2013-02-21

Thank you. I will send you something once I have gathered them. Pretty much here is how I understand it in terms of multipathing for an HA.

Controller 1, connects to:

Channel A, IN port of first shelf of stack

Channel B, OUT port of the last shelf of stack

Controller 2 connects to:

Channel B, IN port of first shelf of stack

Channel A, OUT port of last shelf of stack

Is that correct?

bbjholcomb · ‎2013-02-22

You got it.

I would use the same controller ports, Controller 1 connects to stack 1 A IN on controller port 1a and B out would be port 3a on controller 1. Controller 2 would connect to stack 1 A OUT on 1a and port 3a for B IN on the last shelf of the stack 1. To verify this storage show disk -p on both controllers, during downtime you should be able to pull module A on shelf 1 stack 1 and both systems should stay up, you will see errors messages after a few minutes. Re-insert the module after a few minutes and check the MPHA again.

DUANESTRYDER · ‎2013-02-23

Thanks again. six of our stacks are saying " primary and redundant are on the same path" Just by running show disk -p, I can see that they are either both on A or both on B module. On top of that both cables are connected to the same ASIC on the controller (example: 1a, 1b) So they are two issues it seems. What do you think is the best approach in terms in order in which of these issue I should first? should I separate module connectivity first or the ASIC connectivity?

Thanks,

md

bbjholcomb · ‎2013-02-24

Since you have 4 cards in the controller you can use (assuming 3 stacks are on the A controller and 3 stacks are on controller B), you can connect them like this:

(I assume stacks 1-3 are assigned to controller A and stack 4-6 are assigned to controller B)

1a/3a stack1

1b/3b stack2

1c/3c stack3

4a/11a stack 4

4b/11b stack 5

4c/11c stack 6

You can't use 1d/1c on stack for two reasons, same ASIC and what would happen if card in slot 1 failed.

If you have any any stacks were the primary port on the shelf if the same such as:

PRIMARY PORT SECONDARY PORT SHELF BAY

-------- ---- --------- ---- ---------

2a.06.0 A 1a.06.0 A 6 0

You should be able switch the cables on the last shelf in stack to correct this problem. It would be better to do this doing downtime, it shouldn't cause any down time, after each change check it.

Once you get the disk stacks connected properly:

PRIMARY PORT SECONDARY PORT SHELF BAY

-------- ---- --------- ---- ---------

2a.06.0 B 1a.06.0 A 6 0

Once you get the disk stacks connected properly first, you should be able to move the connections on the controller. You might want to do at separate times. You should do this during a scheduled downtime and let your end users know that an outage is possible. I would consider using NetApp Professional Services to plan and do this, six stacks is a lot to take on, if you make a mistake or something else goes wrong while you are doing this you could lose data. You could take down the cluster, boot in maintenance mode, make the changes and check it in maintenance to be safe. This will allow you check it with "storage show disk -p" and "aggr status -r". Lots of planning, verify and verify again to be sure.

DUANESTRYDER · ‎2013-02-28

Thanks again, you've been really great help. I learned alot.

md

bbjholcomb · ‎2013-03-01

I hope I was able to help, good luck

DUANESTRYDER · ‎2013-02-23

Thanks again. also I guess another question is when one path is disconnected, the redundant takes over, but is there a setting that dictates how this happens, meaning some option that has to be enabled? or timeout that has to be configured? things of that nature that I should be aware of. I know before in another more stable environment, I added a shelve in a DS4243 stack and everything went smoothly without disruption. But that is SAS connection, I don't know if variables are different with a FC HBA shelves.