Mixed sas stack 4xDS4486-7k and 2xDS2246-10k

[Mattb]

I've read the previous posts about mixed SAS stacks and I am well aware of the homogeneous stack limits per shelf type

SSD (4 shelf per stack)

DS 4486 (5 shelf per stack)

Anything else (10 shelf per stack)

The DS4486 has double disks so each one uses up 2 SAS versus 1 SAS which completely explains the 5 shelf limit for Tandem shelves

So with all this well in mind I built a stack like this:

4xDS4486 (thinking ok 2 SAS per equals 8 SAS used)

I should have room for 2 more normal DS2246 with 10k drives bringing my stack to a full 10.

I built it out and when adding the last of the 2 DS2246 I received the ontap alert that the stack has 6 shelves but SHOULD only have 5...

It is almost as if having just 1 DS4486 in a stack seems to make the system think EVERY shelf in that stack is a tandem and primitivley applies the 5 tandem shelf limit

It seems to me I would get the same error with a stack built like this:

1 DS4486 with only 7k

5 DS2246 with only 10k

Is there anyway to bypass this limit? Is it a hard limit? Meaning in spite of the alert could I still use that last DS2246's 10k disks? (The 6th shelf)

I went ahead and removed the last DS2246 so now I just have the 4xDS4486 and 1xDS2246...

I have zero qualms running 4xDS4486 with 7k and 2xDS2246 with 10k and it should calculate to a limit of 10 SAS ID's( 2x4=8 + 1x2=2 ==10)

Appreciate any insight by those with experience in these matters

FAS8040a HA pair running 8.2.5p5 7mode

[TMACMD]

What drives in the 4486? you know the 3T are no longer supported by NetApp and the 4T go end of support at the end of OCT?

I am reading HWU like this:

DS4486 -> Max Stack size of 5

You have 4 x DS4486 and then one more shelf and you hit the max.

It is like a "least common denominator" thing.

Sure it says ten for the other shelves, but it you attach the 4486, it will likely drop the stack back to 5.

Regarding the DS4486 and other shelves: Intermixing shelves is supported but not recommended.

Get another SAS card and make a new stack.

[Mattb]

2 of the DS4486 have 8T and the other 2 have 6T.

The DS2246 have 600GB 10k

Cant get another SAS card and make a new stack. The FAS8040a supports 3 stacks. I have 1 dedicated SSD stack, one dedicated 10k stack with 10 ds2246. And this last stack with 4xDS4486 and 1 ds2246...since the second ds2246 causes an alert.

It sounds like you are saying yes...even adding a single ds4486 to any stack will immediately and primitivley gimp the stack to 5 shelves and ignorantly assume that all shelves in that stack are tandem type. I am not aware of any guidance saying its ok but your really shouldn't have standard shelves with spinning disks in the same stack as tandem shelves

Does anyone know of an option to disable the alert or override it?

[TMACMD]

I didn't design the technology.

There may be some technological reason for the limiting especially with the design of that particular shelf. Remember that shelf, when a drive fails, the second in the same carrier is pre-failed and there is logic to try and prevent someone from placing two drives in the same carrier into the same raid group.

And to your point of "The FAS8040a supports 3 stacks".

One stack is defined as a SAS-port pair, like 0a and 1d.

If you have just the ONBOARD ports, you have: 0a/0d and 0c/0b. (two stacks)

With a single card, I get: 0a/1d, 1a/0d, 0c/1b and 1c/0b (four stacks)

Each card added is another pair of stacks. Not sure how your are getting 3 stacks unless by some shot you are one of the few actually using a locally attached SAS tape library/drive.

All your shelves are 6G SAS. All you need are 4 QSFP-SAS cables long enough to reach the ends of your 4th stack.

[Mattb]

Ah interesting...ok On our 8040a with 3 stacks. Here is the detail

HA pair

A head has 2xquad port SAS hbas. (1 onboard and 1 pcie)

B head has 2xquad port SAS hbas (1 onboard and 1 pcie)

Ssd stack(#1) is using

--Head A, HBA 1, port C

--Head A, HBA 2, port A

--Head B, HBA1, port C

--Head B, HBA2, port A

10k stack (#2) is using

--Head A, HBA 1, port A

--Head A, HBA 2, port C

--Head B, HBA1, port A

--Head B, HBA2, port C

7k ds4486/10kds2246 stack (#3) is using

--Head A, HBA 1, port D

--Head A, HBA 2, port D

--Head B, HBA1, port D

--Head B, HBA2, port D

So with the above I see that as 3 stacks. 

I have 4 more SAS HBA ports free...the B ports that are unused.

Are you thinking I can add a 4th stack? Have I misunderstood the 3 stack limit for 8040a ha pair?

I have run config advisor with all the extended health checks and the filer is 100% clean...all stacks are multipath no warnings.

[TMACMD]

Oye. That is actually not cabled to best practices!

You can certainly plug into all four remaining B ports. I would try to clean up a bit.

You can think of it this way: 

From Filer:

OUT ports (go to the LEFT/square IOM ports) are A and C,

RETURN ports (come from the RIGHT/circle IOM ports)are B and D.

This is how I would be cabled:

Node 1/0a -> shelf 10, IOMA, Square(left)

Node 2/0a -> shelf 10, IOMB, Square(left)

Node 1/1d -> shelf 11, IOMB, Circle/right)

Node 1/1d -> shelf 11, IOMA, Circle/right)

Node 1/1a -> shelf 20, IOMA, Square(left)

Node 2/1a -> shelf 20, IOMB, Square(left)

Node 1/0d -> shelf 21, IOMB, Circle/right)

Node 1/0d -> shelf 21, IOMA, Circle/right)

Node 1/0c -> shelf 30, IOMA, Square(left)

Node 2/0c -> shelf 30, IOMB, Square(left)

Node 1/1b -> shelf 31, IOMB, Circle/right)

Node 1/1b -> shelf 31, IOMA, Circle/right)

Node 1/1c -> shelf 40, IOMA, Square(left)

Node 2/1c -> shelf 40, IOMB, Square(left)

Node 1/0b -> shelf 41, IOMB, Circle/right)

Node 1/0b -> shelf 41, IOMA, Circle/right)

With a single card, I get: 0a/1d, 1a/0d, 0c/1b and 1c/0b (four stacks)

More than anything to limit SAS failure domains.

[Mattb]

I thought you might make mention of that. : ) I am somewhat surprised that config advisor doesnt complain. I'm totally open to recabling them if I can do it without disrupting the data service on the aggregates

But can you explain how it makes sense with the (3 SAS stack limit)? Each stack has 4 SAS ports going to the heads and you said earlier that a stack is defined by a sas port pair. So if in 1 stack I'm using A C and in the second stack I am using C and A and the 4th is D and D. Isnt that still 3 stacks?

If I recable as you suggest to get to A and D, A and D, C and B, C and B...are you saying since I would only be using the A and D pairs and C and B pairs that it would only counts as 2 stacks even though I have 4 actual stacks due to the pair combinations?

If this is all the case and I can actually add what I would consider a 4th SAS stack on my existing HBAs this would be the best news!

[TMACMD]

Older version of CA used to complain. They took it out....I Did not like that. If your system is reporting to Active IQ,  I would bet it is complaining there 😁 I have a lot of customers that are air-gapped and cannot report so when I cable I need to make sure myself it is done to proper spec.

Port-Pair is defined per Node. So port pair 0a/1d, we use the same ports on both nodes, like this:

Both 0a ports go the same shelf and same port (square port), but different IOMs, node 1 to A and node 2 to B

Node 1/0a -> shelf 10, IOMA, Square(left)

Node 2/0a -> shelf 10, IOMB, Square(left)

Both 1d ports go the same shelf and same port (circle port), but opposite IOMs, node 1 to B and node 2 to A

Node 1/1d -> shelf 11, IOMB, Circle/right)

Node 1/1d -> shelf 11, IOMA, Circle/right)

To be fair/honest, I think there are still a few bugs in 7-mode (less so in newer version of ONTAP) that when you re-cable a shelf stack HOT, you may hit some issue that will eventually panic. To alleviate, anytime you make changes. always wait at least 30 second before re-attaching the SAS cables. Also, be sure to do frequent takeover/givebacks to clear memory to avoid hitting issues above.

Of course, if you can get an outage and are able to halt both nodes, re-cable then.

Or, do a takeover, after the taken-over node starts to boot, interrupt it. Re-cable appropriately.  Boot into maintenance mode. You should be able to "sysconfig -av" and still Multi-Path HA near the top of the output. Also, you can use "sasadmin expander_map". On one path you should see IOM-A and the shelf count go up (or possibly down) and on the other path IOM-B, the count should go down (or up if the other went down).

If all looks good, halt the node and then boot....giveback and then repeat.

Of course, if you happen to be using the wrong IOM ports, then, you need to shut down 😐

[Mattb]

I appreciate that info on limiting disruption during the SAS moves and staying on top of it.

We are going to do this if we can build out a 4th stack and its a gamechanger for us in another way

I think on the previous 2 posts there are some typos on the 4th connection in the pairs where you outline your recommendation. it looks like it should be node 2. Like below with the correction

Node 1/0a -> shelf 10, IOMA, Square(left)

Node 2/0a -> shelf 10, IOMB, Square(left)

Node 1/1d -> shelf 11, IOMB, Circle/right)

Node 12/1d -> shelf 11, IOMA, Circle/right)

I am really amazed that we can move the SAS head ports on the stacks around and by co-locating the port-pair and using the same "0a" and "1d" on both heads for Stack 1 and then ALSO able to use "0d" and "1a" on both heads for Stack 2 and they both shakeout to a SAS stack HWU cost of -1 because its 1 port pair ("a" and "d") driving 2 actual stacks of SAS. Does that sound right?

So I think this is the correct/final of what you are saying

(Stack 1)

Node 1/0a -> shelf 10, IOMA, Square(left)

Node 2/0a -> shelf 10, IOMB, Square(left)

Node 1/1d -> shelf 11, IOMB, Circle/right)

Node 2/1d -> shelf 11, IOMA, Circle/right)

(Stack 2)

Node 1/1a -> shelf 20, IOMA, Square(left)

Node 2/1a -> shelf 20, IOMB, Square(left)

Node 1/0d -> shelf 21, IOMB, Circle/right)

Node 2/0d -> shelf 21, IOMA, Circle/right)

Node 1/0c -> shelf 30, IOMA, Square(left)

Node 2/0c -> shelf 30, IOMB, Square(left)

Node 1/1b -> shelf 31, IOMB, Circle/right)

Node 2/1b -> shelf 31, IOMA, Circle/right)

Node 1/1c -> shelf 40, IOMA, Square(left)

Node 2/1c -> shelf 40, IOMB, Square(left)

Node 1/0b -> shelf 41, IOMB, Circle/right)

Node 2/0b -> shelf 41, IOMA, Circle/right)

So when counting the above I would count the first 2 stacks as together..."using up" 1 single SAS port pair, namely a/d.

The third and fourth stacks would count together as..."using up" another single SAS port pairs

bringing the total to 2 SAS port pairs in terms of HWU limit of 3 SAS stacks per 8040a HA head

Wow again this is a game changer because it means we could even add in additional HBA's and potentially drive a third port pair that would yield what we could consider 2 more SAS stacks for a total of "what we would consider" 6 stacks. 

If you can confirm this understanding, I will close this question with much gratitude and thanks

[TMACMD]

A port-pair is per node(0a/1d), but should be the same port pair between HA-pairs.

(node 1 0a/1d and node 2 0a/1d)

The port-pairs used on both controllers is the stack. (stack 1)

In your hardware you have 4 port-pairs per node which allows for 4 stacks:

0a/1d, 1a/0d, 0c/1b and 1c/0b (four stacks)