Active/Active cluster cabling

KemDatacenter · ‎2011-10-08

We have NetApp Active/Active cluster already configured (by outside organization) with two FAS3020 filers, two SAS and two SATA shelves. Each Filer owns one SAS and one SATA shelf. SAS shelves have ESH2 modules, SATA shelves have AT-FCX modules. I'm trying to understand its cabling. Active/Active Configuration Guide says cabling for standard active/active configuration should be as follows:

FAS1 0a to shelf SAS1 A-In

shelf SAS1 A-Out to shelf SATA1 A-In

on the SATA1 terminate switch to on

FAS1 0b to shelf SAS2 B-In

shelf SAS2 B-Out to shelf SATA2 B-In

on the SATA2 terminate switch to on

FAS2 0a to shelf SAS2 A-In

shelf SAS2 A-Out to shelf SATA2 A-In

on the SATA2 terminate switch to on

FAS2 0b to shelf SAS1 B-In

shelf SAS1 B-Out to shelf SATA1 B-In

on the SATA1 terminate switch to on

In fact we have following cabling:

FAS1 0a - SAS1 A-In

FAS1 0b - SATA1 A-In

FAS1 0c - SAS2 A-In

FAS1 0d - SATA3 A-In

FAS2 0a - SAS2 B-In

FAS2 0b - SATA1 B-In

FAS2 0c - SAS2 B-In

FAS2 0d - SATA3 B-In

And I don't see any "terminate switches" there.

Can somebody explain me what type of configuration we have? It doesn't conform to what documentation says.

Here is the ouput from Filer 1:

fas1> storage show disk -p

PRIMARY PORT SECONDARY PORT SHELF BAY

------- ---- --------- ---- ---------

0a.16 B 1 0

0a.17 B 1 1

0a.18 B 1 2

0a.19 B 1 3

0a.20 B 1 4

0a.21 B 1 5

0a.22 B 1 6

0a.23 B 1 7

0a.24 B 1 8

0a.25 B 1 9

0a.26 B 1 10

0a.27 B 1 11

0a.28 B 1 12

0a.29 B 1 13

0b.48 B 3 0

0b.49 B 3 1

0b.50 B 3 2

0b.51 B 3 3

0b.52 B 3 4

0b.53 B 3 5

0b.54 B 3 6

0b.55 B 3 7

0b.56 B 3 8

0b.57 B 3 9

0b.58 B 3 10

0b.59 B 3 11

0b.60 B 3 12

0b.61 B 3 13

0c.32 A 2 0

0c.33 A 2 1

0c.34 A 2 2

0c.35 A 2 3

0c.36 A 2 4

0c.37 A 2 5

0c.38 A 2 6

0c.39 A 2 7

0c.40 A 2 8

0c.41 A 2 9

0c.42 A 2 10

0c.43 A 2 11

0c.44 A 2 12

0c.45 A 2 13

0d.64 A 4 0

0d.65 A 4 1

0d.66 A 4 2

0d.67 A 4 3

0d.68 A 4 4

0d.69 A 4 5

0d.70 A 4 6

0d.71 A 4 7

0d.72 A 4 8

0d.73 A 4 9

0d.74 A 4 10

0d.75 A 4 11

0d.76 A 4 12

0d.77 A 4 13

And from Filer 2:

FAS2> storage show disk -p

PRIMARY PORT SECONDARY PORT SHELF BAY

------- ---- --------- ---- ---------

0a.16 A 1 0

0a.17 A 1 1

0a.18 A 1 2

0a.19 A 1 3

0a.20 A 1 4

0a.21 A 1 5

0a.22 A 1 6

0a.23 A 1 7

0a.24 A 1 8

0a.25 A 1 9

0a.26 A 1 10

0a.27 A 1 11

0a.28 A 1 12

0a.29 A 1 13

0b.48 A 3 0

0b.49 A 3 1

0b.50 A 3 2

0b.51 A 3 3

0b.52 A 3 4

0b.53 A 3 5

0b.54 A 3 6

0b.55 A 3 7

0b.56 A 3 8

0b.57 A 3 9

0b.58 A 3 10

0b.59 A 3 11

0b.60 A 3 12

0b.61 A 3 13

0c.32 B 2 0

0c.33 B 2 1

0c.34 B 2 2

0c.35 B 2 3

0c.36 B 2 4

0c.37 B 2 5

0c.38 B 2 6

0c.39 B 2 7

0c.40 B 2 8

0c.41 B 2 9

0c.42 B 2 10

0c.43 B 2 11

0c.44 B 2 12

0c.45 B 2 13

0d.64 B 4 0

0d.65 B 4 1

0d.66 B 4 2

0d.67 B 4 3

0d.68 B 4 4

0d.69 B 4 5

0d.70 B 4 6

0d.71 B 4 7

0d.72 B 4 8

0d.73 B 4 9

0d.74 B 4 10

0d.75 B 4 11

0d.76 B 4 12

0d.77 B 4 13

KemDatacenter · ‎2011-10-08

I've just found following statement in documentation: Both Fibre Channel and SATA storage are supported in standard active/active configurations, as long as the two storage types are not mixed on the same loop.

So my assumption is wrong and I can't cable SAS and SATA together. Documentation also says: Repeat Step 1 to Step 8 for each pair of loops in the active/active configuration, using ports 3 and 4 for the next loop, ports 5 and 6 for the next one, and so on.

Does it mean I should have 4 loops, 1 for each shelf and connect shelves like:

FAS1 0a to SAS1 A-In

FAS1 0b to SATA1 A-In

FAS1 0c to SAS2 B-In

FAS1 0d to SATA B-In

FAS2 0a to SAS1 B-In

FAS2 0b to SATA1 B-In

FAS2 0c to SAS2 A-In

FAS2 0d to SATA A-In

It also doesn't match to what we have.

aborzenkov · ‎2011-10-08

You have four loops. Whether head is connected to channel A or channel B is completely irrelevant as long as you keep it consistent and it suits you. In the times of hardware based disk ownership and single path connections I tended to connect one head to channel A and another head to channel B, exactly as had been done in your case.

Personally I’d rather implement MP-HA by creating 2 stacks, one FC and one SATA, and enable software based disk ownership. It needs the same amount of FC ports but gives better resiliency. But it may require additional equipment (cables and/or SFP).

aborzenkov · ‎2011-10-08

FAS3020 does not support SAS shelves. You probably mean FC.

There is no terminate switches since ESH2 modules.

You have normal single path connection. I would do it differently but this is working configuration.

KemDatacenter · ‎2011-10-08

Can you point me to documentation section where it's described? As far as I understand Filer 1 needs to be connected to A-In port of shelves it owns and B-In for partner shelves. And vice versa for Filer 2.

aborzenkov · ‎2011-10-09

Actually now when I looked more closely on “storage disk show -p” output, half of FC ports on each head are connected to channel A and half of ports are connected to channel B.

And sorry for confusion, you are right. For hardware based disk ownership (default on FAS3000) disk ownership is indeed determined by channel; filer connected to channel A owns all disks in a shelf.

I have worked with software based disk ownership for so long that I tend to forget it. ☺

KemDatacenter · ‎2011-10-09

I just entered "storage show" and it doesn't show "SANOWN not enabled". According to documentation it means system uses software owhership. (In fact it doesn't also show SANOWN enabled. Does it have to?)

Right, if "storage disk show -p" shows that half of FC ports are connected to A and half to B then I'm wrong again in assumption that each filer is connected to it's own channel. Tomorrow I'll drive to DC and watch it more closely.

Can you also explain Loop A and Loop B concept? From what I understand if I connect Filer FC port to In module port of any shelf and don't connect Out port of this module to In port of next shelf then it comprises a "Loop". It means I can have number of loops less or equal to number of Filer FC ports. Then why does NetApp documentation always speaks about two loops: Loop A and Loop B? Does it mean that if I connect four Filer FC ports to four A-In ports on disk shelves then Filer build single A Loop inside itself?

aborzenkov · ‎2011-10-09

OK, so you do have software based disk ownership. In this case my original statement holds true – it does not matter (technically) how you connect shelves, filer always finds own disks. The only consideration is consistency and ease of maintenance.

In case of SANOWN enabled nothing should be displayed. You can verify it be checking whether command “disk show” exists at all. If SANOWN is disabled, this command is disabled as well.

“Loops” in this context means all shelves connected to an FC port. You can daisy chain multiple shelves, supported number is 6 on single loop. With advent of MP-HA and SAS notion of “loop” became a bit blurry, so what is often used today (in NetApp documentation) is “stack”. It has the same meaning – it means shelves which are daisy chained and connected to the same filer adapter(s).

Each adapter forms a single loop. Each shelf has two channels – channel A and B. With single path connection you connect FC port on filer to channel A (although as mentioned with SANOWN it does not really matter). With multipath connection you connect one FC port on filer to channel A and another FC port on the same filer to channel B. So you have two “loops” A and B connected to the same “stack” of shelves. Each stack has own loops, connected to different FC ports.

In your case you have 4 loops, forming 4 single path stacks each consisting of single shelf.

KemDatacenter · ‎2011-10-09

Thanks for answers, now it's much more clear.

Am I understanding correctly that Loop A and Loop B terminology has to do only with shelves and when I speak about filers I should say e0a loop, e0b loop, etc?

aborzenkov · ‎2011-10-09

I briefly reviewed Active/Active guide and I do not see where they mention “Loop A” at all ☺ They speak about connecting adapters to shelf channel A or B. But you are right in that A or B applies to shelf.

KemDatacenter · ‎2011-10-09

Right, I took a look into documentation and they use "Channel A" much more often. But sometimes they say "Loop A" and LED on shelf bezel also says "Loop A".

KemDatacenter · ‎2011-10-09

And one more question. You said you'd rather implement MP-HA. How can you do that if there are only two In ports on each shelf and Out ports can be used only to connect more shelves to the same loop? You can etiher use these two In ports for MP or HA.

aborzenkov · ‎2011-10-09

You can and this is described in Active/Active guide. You connect one adapter to In port on channel A on one end of stack and another adapter to Out port on channel B on another end of stack.

KemDatacenter · ‎2011-10-09

You're right again. I found it.

KemDatacenter · ‎2011-10-09

Спасибо за короткую лекцию по подключению NetApp. Hope I won't be banned for Russian.