Solved: Re: Performance issue on 3240 HA 7mode

larryblanco · ‎2014-01-02

I have a ticket open with Netapp but wanted to get feedback on what is to be expected performance.

We recently spent some bucks on getting a SSD shelf with 24x 200GB SSD's for our 3240 pair. The SSD are the Samsung eMLC ones. We're running 8.1.2p3

We can't seem to get any real performance out of them.

Iops wise, we are only able to max out at about 35-40K. I would of expected at least 80K+

I'm using IOMETER with: 4K transfer - 100% Random distribution - 100% Read - 4K aligned IO

Then trying to tranfer 16MB chunks we get about 380MB/s transfer rates. 22-23 iops. I would of expected a bit more. atleast 600MB writing to the rg.

it's a 23 disk RG running raid DP.

The box uses iscsi and NFS. In this instance, I am accessign the disks via a VMWare VM using the MS initiator. I've tried RDM's, and NFS. RDM performance is pretty par with and MS initiator within the VM..

NFS suffers from even worse performance.

The network is 10Gb. Inside the esxi boxes i have the Intel x520DA nic and Cisco Nexus 5548 switches conenction the boxes to the filers.

Any thoughts on this would be helpful. So far, after sending in some perfstats, NetApp tech support has stated that they don't see anything out of the ordinary.

I say there isn't anything out of the ordinary. It just that we are not seeing the performance we should see out of the disks.

Again, any help would be greatly appreaciated.

Larry

rmatsumoto · ‎2014-01-06

That bottom sysstat -M 2 output and some of the output in the middle suggest to me that the task is CPU-bound, which is consistent with how our FAS32xx perform in our environment. When you say that the aggr is a 23-disk RG, is it a regular aggr with nothing but SSDs, or is it a flashpool aggr? Either way, the delay is somewhere beyond your disks, based on what you posted. Right now I'd lean towards the head itself(at least for some of those tests).

View solution in original post

radek_kubka · ‎2014-01-03

Hi Larry,

I'm not saying it will solve anything and you might have looked into that already, but can you run "sysstat -x 2" whilst doing throughput tests & post the results here?

Maybe CPU gets hammered so heavily that it becomes the bottleneck?

Regards,

Radek

larryblanco · ‎2014-01-03

Normal output without any load testing from IO meter.

sysstat -x 2

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

22% 2213 57 0 3694 12493 12161 10472 12 0 0 0s 92% 0% - 12% 0 0 1424 0 0 5644 1642

32% 2247 34 0 4457 12787 80785 9498 16 0 0 1 98% 0% - 11% 357 0 1819 0 0 5125 72024

33% 2089 61 0 4284 11490 71957 9942 0 0 0 0s 98% 0% - 11% 248 0 1886 0 0 5203 61249

28% 1566 22 0 4634 22631 46895 8652 12 0 0 1 98% 0% - 7% 760 0 2286 0 0 16354 36091

44% 3079 5 0 5170 11831 50114 32710 11490 0 0 1 98% 33% Tf 16% 3 0 2083 0 0 3593 37095

34% 5582 17 0 6510 22378 26915 25053 47656 0 0 58s 95% 100% :f 23% 195 0 716 0 0 4167 16564

27% 5510 2 0 6563 7442 20225 10656 51356 0 0 58s 96% 100% :f 14% 624 0 427 0 0 1495 7045

28% 7703 5 0 8060 6061 17982 7216 27056 0 0 0s 94% 50% : 10% 225 0 127 0 0 91 8389

29% 4417 9 0 5044 15395 56754 8402 0 0 0 52s 98% 0% - 7% 44 0 574 0 0 558 46404

25% 2059 27 0 3319 8529 76749 8176 16 0 0 0s 98% 0% - 7% 5 0 1228 0 0 226 63087

34% 2252 20 0 3256 9665 81286 12370 12 0 0 55s 97% 0% - 17% 204 0 780 0 0 3028 61723

40% 3320 5 0 4152 19817 92174 8836 4 0 0 49s 98% 0% - 15% 3 0 824 0 0 10454 73855

43% 1970 8 0 2610 9599 59711 46786 59622 0 0 46s 97% 50% Tf 29% 0 0 632 0 0 4496 46047

23% 2824 0 0 3076 5056 42455 19338 47264 0 0 46s 96% 100% :f 25% 1 0 251 0 0 162 21742

28% 4752 3 0 5101 7774 57057 11126 20228 0 0 1s 97% 55% : 20% 3 0 343 0 0 175 33071

28% 3462 6 0 4665 31596 52672 13554 12 0 0 1s 96% 0% - 22% 0 0 1197 0 0 23282 29078

26% 5244 3 0 5618 11236 38894 11504 16 0 0 0s 95% 0% - 18% 3 0 368 0 0 134 19120

22% 3690 3 0 3866 5281 54532 34416 0 0 0 51s 93% 0% - 22% 0 0 173 0 0 117 18100

26% 3771 6 0 4812 6211 58189 15632 12 0 0 44s 96% 0% - 20% 709 0 326 0 0 132 31613

Load output IO Meter: Transfer Request Size 4K - 100% Random Access - 100% Access Specification - 100% Read Distribution - 4K Aligned IO

sysstat -x 2

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

87% 3494 11 0 43814 16150 205181 127896 4 0 0 7s 45% 0% - 41% 2 0 40307 0 0 6511 189587

86% 2931 20 0 42097 13529 219815 122402 12 0 0 6s 50% 0% - 40% 0 0 39146 0 0 6315 205359

88% 3699 162 0 41929 8958 230617 119240 16 0 0 6s 58% 0% - 40% 3 0 38065 0 0 146 213639

88% 2525 54 0 38680 6422 227492 120333 637 0 0 6s 60% 11% Tn 40% 0 0 36101 0 0 188 215957

88% 2678 47 0 40523 6426 195799 144529 51259 0 0 6s 50% 100% :f 41% 47 0 37751 0 0 142 177521

88% 4158 66 0 43367 11776 196611 136788 35372 0 0 7s 49% 100% :f 41% 3 0 39140 0 0 1969 179757

87% 3595 31 0 41575 20804 173151 117057 27068 0 0 0s 44% 82% : 40% 11 0 37938 0 0 4732 157339

87% 2551 1 0 43810 27262 183591 120958 16 0 0 7s 46% 0% - 40% 50 0 41208 0 0 19916 170101

89% 3288 15 0 43064 14393 225702 116794 12 0 0 6s 56% 0% - 38% 16 0 39745 0 0 2820 213647

89% 2617 14 0 42945 13388 219565 118542 0 0 0 6s 51% 0% - 40% 71 0 40243 0 0 4174 205255

89% 3873 3 0 40678 13350 250964 113674 16 0 0 6s 61% 0% - 40% 5 0 36797 0 0 4759 231133

88% 2821 76 0 38778 10641 183914 126484 7266 0 0 6s 55% 18% Tn 40% 13 0 35868 0 0 2960 166863

88% 2579 0 0 39427 21474 165066 133378 57494 0 0 6s 46% 100% :f 41% 4 0 36844 0 0 12661 151652

87% 1904 34 0 43533 7382 184509 129775 45741 0 0 6s 38% 100% :f 44% 0 0 41595 0 0 193 173270

88% 6506 23 0 46367 7884 178743 122516 8570 0 0 6s 42% 26% : 40% 0 0 39838 0 0 313 166676

89% 9307 26 0 43438 8633 229781 103372 4 0 0 6s 60% 0% - 35% 2 0 34103 0 0 502 212554

88% 2742 1 0 37510 7216 275261 108126 12 0 0 7s 66% 0% - 36% 30 0 34737 0 0 406 262300

88% 3928 21 0 45354 8067 209393 121533 16 0 0 7s 45% 0% - 42% 16 0 41389 0 0 183 195308

87% 2885 2 0 46181 7296 193702 130646 0 0 0 6s 39% 0% - 44% 19 0 43275 0 0 161 177334

Load output IO Meter: Transfer Request Size 16MB - 100% Sequential Access - 100% Access Specification - 100% Read Distribution - 4K Aligned IO

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

68% 4672 0 0 7325 8923 354433 265440 43280 0 0 18s 99% 100% :f 40% 277 0 2376 0 0 138 342716

59% 3335 15 0 5369 6449 344757 217788 32070 0 0 19s 99% 100% :f 35% 53 0 1966 0 0 254 335380

57% 3500 0 0 4901 7972 317931 248322 7686 0 0 0s 99% 35% : 21% 0 0 1401 0 0 173 307724

54% 3134 27 0 4504 4627 317465 260380 12 0 0 0s 99% 0% - 20% 4 0 1339 0 0 233 296178

68% 4477 1 0 6348 6669 388156 267822 0 0 0 16s 98% 0% - 18% 0 0 1870 0 0 735 345821

63% 5636 1 0 7556 10006 322534 240190 16 0 0 13s 98% 0% - 22% 139 0 1780 0 0 2289 297738

61% 3532 4 0 6155 8938 341386 272162 12 0 0 0s 99% 0% - 14% 0 0 2619 0 0 3211 320131

74% 4689 8 0 7379 37665 304690 262102 34879 0 0 10s 98% 64% Tf 37% 0 0 2682 0 0 20509 282759

61% 3424 3 0 5363 4557 341655 223644 40968 0 0 0s 99% 100% :f 28% 3 0 1933 0 0 151 329314

64% 8645 1 0 10877 7107 340460 237286 8582 0 0 >60 99% 32% : 17% 163 0 2068 0 0 141 326623

68% 9420 1 0 11683 8492 369522 238942 4 0 0 10s 99% 0% - 19% 106 0 2156 0 0 204 351236

72% 5876 1614 0 10152 10516 342719 273166 12 0 0 0s 98% 0% - 19% 391 0 2271 0 0 3183 317886

83% 7691 2922 0 12396 11209 316906 255950 16 0 0 0s 98% 0% - 19% 247 0 1536 0 0 2116 300319

75% 8573 89 0 10173 47169 319148 256336 0 0 0 13s 98% 0% - 16% 34 0 1477 0 0 13296 275942

78% 7118 7 0 8790 25555 246300 251986 37842 0 0 12s 98% 68% Tf 29% 283 0 1382 0 0 17348 235860

72% 9251 20 0 10711 22171 291877 290006 52854 0 0 11s 98% 100% :f 26% 3 0 1437 0 0 12895 278790

75% 7697 36 0 9586 20737 344585 246794 43434 0 0 0s 98% 100% :f 34% 0 0 1853 0 0 11778 328419

72% 8014 7 0 10487 25278 321429 266278 7918 0 0 14s 98% 33% : 28% 0 0 2466 0 0 15352 306345

74% 8510 6 0 10168 29228 298529 285692 12 0 0 13s 98% 0% - 24% 3 0 1649 0 0 15079 287470

75% 7305 0 0 8715 20279 277713 247068 33339 0 0 12s 98% 41% Ms 26% 0 0 1410 0 0 12825 260506

timo_puronen · ‎2014-01-03

Have you applied best practices on both - the shelf cabling and vmware configuration? Also, go over 10gig network configuration, I have experienced issues between storage and ESXi hosts in 10gig ethernet. VSC helps somewhat but, performance tuning in 10gig ethernet always requires some effort. SSD shelf should be cabled in separate stack or at least be the first one in stack.

Best regards,

Timo

larryblanco · ‎2014-01-03

We had another SAS controller installed on the 3240 for the SSD. We cabled up the SSD shelf to it. I also added another SAS shelf and cabled it to the SAS controller to it. The SAS shelf current sits empty. I only have data on the SSD's at this time.

The esxi server are cabled correctly to the switches. I have verified throughput. I can saturate the each link to about 9.5GBps. I removed any network control IO for testing of the ESXi servers. I can vmotion a server from one ESXi server to another, in record time. I can see the nics transferring between servers and hitting 8+ GBps. Granted it goes from the RAM on one server to the RAM of another, but it verifies that the network can handle and put through a lot of data.

I've tweaked nics. Flowcontrol off, jumbo frames, receive windows sizes. U name it.

Switches have MTU set to 9216, They do not report any error frames, no re-transmits, Nothing. Clean as a whistle.

Larry

dburkland · ‎2014-01-03

Can you also send the output of the "sysstat -M 2" command? You will need to run the "priv set diag" command in order to access the "-M" sysstat argument.

Thanks,

Dan

larryblanco · ‎2014-01-04

Dan below are the reading for the same exact load scenarios above.

Normal output without any load testing from IO meter.

sysstat -M 2

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

46% 15% 4% 1% 22% 21% 25% 23% 20% 24% 0% 0% 2% 1% 4% 9% 23%( 20%) 0% 0% 0% 3% 8% 14% 4435 0%

39% 12% 4% 1% 18% 18% 20% 19% 16% 18% 0% 0% 1% 1% 2% 8% 20%( 16%) 0% 0% 0% 4% 6% 14% 4374 0%

55% 21% 8% 4% 27% 25% 28% 27% 28% 27% 0% 0% 1% 2% 3% 14% 35%( 25%) 2% 0% 0% 4% 7% 13% 5973 4%

61% 39% 28% 16% 39% 39% 40% 39% 38% 22% 0% 0% 5% 10% 3% 11% 67%( 38%) 7% 0% 0% 17% 6% 9% 4340 93%

17% 5% 1% 0% 9% 10% 11% 9% 5% 8% 0% 0% 1% 1% 2% 2% 7%( 7%) 0% 0% 0% 3% 4% 6% 1023 0%

31% 12% 4% 1% 15% 16% 18% 16% 10% 15% 0% 0% 1% 1% 3% 2% 19%( 15%) 0% 0% 0% 6% 4% 9% 2101 0%

26% 9% 4% 1% 13% 13% 15% 13% 8% 12% 0% 0% 1% 1% 2% 2% 16%( 12%) 0% 0% 0% 5% 4% 8% 2303 0%

21% 6% 1% 0% 10% 11% 11% 10% 5% 10% 0% 0% 1% 1% 1% 2% 9%( 8%) 0% 0% 0% 3% 5% 7% 2152 0%

25% 10% 5% 2% 14% 14% 15% 16% 11% 10% 0% 0% 1% 2% 1% 4% 18%( 13%) 4% 0% 0% 2% 4% 10% 2498 5%

40% 18% 8% 4% 20% 21% 22% 20% 15% 11% 0% 0% 4% 6% 2% 4% 26%( 18%) 4% 0% 0% 10% 5% 7% 2166 100%

35% 15% 8% 4% 19% 21% 22% 21% 13% 14% 0% 0% 2% 4% 2% 4% 24%( 14%) 0% 0% 0% 10% 4% 11% 2310 100%

34% 10% 3% 1% 15% 15% 17% 15% 13% 17% 0% 0% 1% 1% 1% 5% 19%( 15%) 0% 0% 0% 4% 4% 8% 4411 4%

59% 24% 7% 1% 29% 31% 34% 32% 20% 34% 0% 0% 1% 1% 5% 6% 30%( 27%) 0% 0% 0% 11% 9% 19% 5179 0%

42% 19% 10% 5% 22% 23% 24% 23% 19% 21% 0% 0% 1% 1% 5% 6% 32%( 22%) 0% 0% 0% 7% 6% 10% 5011 0%

26% 8% 2% 0% 13% 13% 15% 15% 8% 12% 0% 0% 1% 1% 2% 3% 13%( 11%) 0% 0% 0% 4% 5% 11% 1942 0%

33% 11% 3% 1% 16% 17% 19% 17% 11% 16% 0% 0% 2% 1% 2% 4% 17%( 14%) 0% 0% 0% 4% 4% 13% 3190 0%

48% 25% 13% 6% 28% 29% 30% 28% 24% 12% 0% 0% 3% 6% 1% 7% 37%( 25%) 7% 0% 0% 15% 5% 16% 2588 96%

26% 9% 3% 1% 12% 13% 14% 12% 10% 10% 0% 0% 2% 2% 1% 4% 15%( 11%) 0% 0% 0% 6% 3% 7% 2058 63%

26% 8% 3% 1% 12% 12% 14% 12% 10% 12% 0% 0% 1% 1% 2% 4% 13%( 11%) 0% 0% 0% 3% 3% 8% 2545 0%

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

32% 10% 3% 0% 17% 19% 19% 20% 9% 14% 0% 0% 2% 1% 2% 3% 15%( 13%) 0% 0% 0% 7% 5% 18% 2154 0%

26% 9% 3% 1% 14% 13% 17% 16% 9% 14% 0% 0% 1% 1% 1% 3% 15%( 12%) 0% 0% 0% 3% 7% 10% 2739 0%

31% 10% 3% 1% 15% 14% 18% 18% 9% 16% 0% 0% 1% 1% 3% 2% 16%( 13%) 0% 0% 0% 5% 6% 10% 2548 0%

Load output IO Meter: Transfer Request Size 4K - 100% Random Access - 100% Access Specification - 100% Read Distribution - 4K Aligned IO

sysstat -M 2

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

98% 96% 88% 60% 89% 94% 94% 94% 75% 54% 0% 0% 27% 28% 38% 5% 160%( 85%) 4% 0% 0% 21% 8% 12% 37527 9%

100% 96% 87% 59% 88% 90% 93% 94% 75% 55% 0% 0% 26% 28% 38% 9% 153%( 81%) 2% 0% 0% 22% 8% 9% 37439 100%

100% 96% 87% 60% 89% 91% 94% 95% 74% 59% 0% 0% 27% 27% 42% 6% 157%( 83%) 0% 0% 0% 18% 9% 10% 41271 47%

98% 94% 83% 54% 85% 89% 91% 92% 69% 55% 0% 0% 30% 29% 44% 5% 148%( 82%) 0% 0% 0% 14% 7% 10% 42856 0%

96% 87% 69% 40% 77% 81% 80% 81% 63% 48% 0% 0% 26% 25% 39% 9% 128%( 73%) 0% 0% 0% 13% 8% 12% 35268 0%

98% 95% 84% 55% 86% 89% 90% 92% 72% 54% 0% 0% 29% 28% 43% 8% 148%( 81%) 0% 0% 1% 14% 7% 10% 41983 0%

100% 97% 87% 58% 88% 93% 93% 94% 72% 56% 0% 0% 27% 27% 43% 5% 155%( 84%) 0% 0% 0% 21% 8% 9% 41240 0%

100% 96% 87% 59% 88% 92% 92% 93% 74% 46% 0% 0% 25% 30% 34% 7% 154%( 82%) 12% 0% 0% 27% 8% 8% 32501 97%

99% 96% 86% 58% 88% 90% 93% 95% 72% 52% 0% 0% 25% 28% 39% 6% 152%( 82%) 0% 0% 0% 33% 8% 9% 36090 100%

99% 92% 79% 50% 83% 85% 87% 90% 70% 50% 0% 0% 25% 25% 38% 7% 146%( 80%) 0% 0% 0% 21% 8% 11% 35440 37%

97% 89% 74% 45% 79% 84% 84% 85% 63% 51% 0% 0% 26% 25% 40% 6% 136%( 76%) 0% 0% 0% 14% 8% 10% 37979 0%

97% 89% 76% 48% 81% 85% 86% 87% 66% 58% 0% 0% 25% 24% 39% 6% 137%( 77%) 0% 0% 0% 14% 9% 11% 38628 0%

98% 95% 83% 53% 87% 91% 92% 93% 70% 67% 0% 0% 24% 23% 40% 6% 145%( 80%) 0% 0% 0% 18% 9% 14% 40987 0%

99% 95% 86% 58% 88% 92% 93% 93% 73% 63% 0% 0% 26% 25% 43% 7% 150%( 82%) 0% 0% 0% 18% 9% 10% 42561 0%

99% 93% 83% 55% 85% 90% 90% 90% 71% 56% 0% 0% 24% 26% 39% 7% 145%( 80%) 10% 0% 0% 17% 8% 9% 36158 62%

100% 97% 88% 59% 89% 92% 94% 96% 74% 60% 0% 0% 27% 29% 43% 6% 151%( 84%) 0% 0% 0% 20% 9% 12% 40758 100%

100% 96% 86% 57% 87% 91% 92% 94% 72% 57% 0% 0% 27% 28% 45% 6% 150%( 82%) 0% 0% 0% 17% 8% 10% 43335 77%

99% 95% 84% 54% 86% 89% 91% 94% 69% 58% 0% 0% 27% 26% 45% 5% 148%( 84%) 0% 0% 0% 14% 9% 10% 44552 0%

99% 96% 86% 57% 87% 91% 93% 95% 71% 59% 0% 0% 24% 24% 43% 4% 153%( 83%) 0% 0% 0% 24% 8% 10% 40607 0%

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

99% 95% 84% 55% 87% 90% 92% 94% 71% 56% 0% 0% 27% 26% 45% 5% 150%( 84%) 0% 0% 0% 18% 7% 12% 43312 0%

100% 94% 83% 53% 86% 88% 92% 94% 69% 55% 0% 0% 28% 27% 46% 5% 146%( 84%) 0% 0% 0% 14% 9% 13% 44818 0%

98% 95% 84% 56% 86% 89% 91% 92% 71% 51% 0% 0% 26% 27% 41% 6% 152%( 83%) 8% 0% 0% 18% 6% 9% 40446 22%

99% 95% 82% 53% 85% 88% 90% 92% 69% 53% 0% 0% 27% 29% 43% 6% 143%( 80%) 0% 0% 0% 21% 8% 9% 40301 100%

100% 95% 85% 55% 87% 90% 92% 95% 70% 57% 0% 0% 26% 26% 44% 6% 148%( 83%) 0% 0% 0% 21% 9% 10% 41948 47%

98% 95% 83% 51% 86% 90% 93% 93% 70% 55% 0% 0% 27% 26% 46% 4% 146%( 81%) 0% 0% 0% 18% 7% 16% 43190 0%

97% 93% 79% 48% 83% 87% 89% 91% 65% 49% 0% 0% 29% 27% 48% 3% 141%( 76%) 0% 0% 0% 17% 6% 14% 41486 0%

97% 94% 81% 52% 84% 88% 90% 93% 66% 50% 0% 0% 30% 29% 50% 3% 141%( 77%) 0% 0% 0% 14% 8% 12% 44740 0%

98% 93% 80% 51% 83% 87% 89% 92% 65% 50% 0% 0% 30% 27% 50% 3% 143%( 77%) 0% 0% 0% 13% 8% 9% 44529 0%

98% 94% 83% 54% 85% 90% 91% 92% 68% 52% 0% 0% 27% 27% 45% 4% 145%( 79%) 5% 0% 0% 19% 8% 9% 41277 52%

99% 94% 83% 54% 85% 89% 91% 94% 67% 52% 0% 0% 29% 27% 49% 3% 145%( 78%) 0% 0% 0% 19% 8% 9% 43466 31%

99% 95% 83% 55% 86% 90% 91% 92% 70% 53% 0% 0% 28% 26% 48% 4% 148%( 79%) 0% 0% 0% 18% 8% 11% 42859 0%

100% 96% 86% 57% 88% 92% 94% 96% 70% 58% 0% 0% 28% 26% 48% 4% 151%( 80%) 0% 0% 0% 19% 9% 12% 44058 0%

97% 90% 76% 47% 81% 85% 86% 88% 64% 50% 0% 0% 28% 26% 46% 4% 138%( 75%) 0% 0% 0% 13% 8% 11% 41602 0%

99% 95% 82% 53% 85% 88% 91% 93% 67% 50% 0% 0% 30% 28% 52% 3% 146%( 78%) 0% 0% 0% 13% 8% 9% 46371 0%

99% 95% 83% 51% 87% 90% 91% 94% 71% 46% 0% 0% 27% 27% 42% 5% 146%( 79%) 5% 0% 0% 23% 8% 16% 37623 69%

98% 94% 81% 52% 85% 89% 91% 92% 66% 49% 0% 0% 31% 28% 51% 3% 143%( 77%) 0% 0% 0% 15% 6% 12% 45390 30%

99% 95% 83% 54% 85% 89% 92% 94% 67% 53% 0% 0% 29% 27% 50% 3% 147%( 79%) 0% 0% 0% 14% 8% 9% 46035 0%

98% 95% 84% 55% 86% 90% 92% 93% 67% 55% 0% 0% 24% 22% 43% 3% 148%( 79%) 0% 0% 0% 30% 7% 10% 37164 0%

Load output IO Meter: Transfer Request Size 16MB - 100% Sequential Access - 100% Access Specification - 100% Read Distribution - 4K Aligned IO

sysstat -M 2

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

81% 54% 32% 14% 49% 52% 54% 52% 38% 32% 0% 0% 10% 15% 13% 5% 70%( 49%) 5% 0% 0% 27% 7% 12% 1984 87%

82% 53% 27% 10% 47% 50% 52% 52% 34% 37% 0% 0% 10% 12% 14% 3% 66%( 49%) 0% 0% 0% 25% 9% 13% 3036 21%

85% 57% 30% 10% 50% 54% 56% 54% 37% 39% 0% 0% 11% 13% 15% 3% 70%( 52%) 0% 0% 0% 27% 8% 16% 2823 0%

88% 63% 38% 16% 57% 61% 62% 63% 40% 52% 0% 0% 8% 11% 17% 3% 79%( 55%) 0% 0% 0% 29% 11% 17% 6541 0%

86% 56% 31% 12% 52% 54% 56% 59% 37% 49% 0% 0% 7% 10% 15% 4% 70%( 50%) 0% 0% 0% 24% 10% 17% 6688 0%

77% 43% 20% 7% 41% 44% 45% 47% 28% 35% 0% 0% 7% 10% 13% 3% 55%( 43%) 0% 0% 0% 21% 8% 13% 3013 0%

86% 60% 36% 16% 54% 56% 59% 61% 39% 39% 0% 0% 9% 14% 14% 4% 77%( 53%) 5% 0% 0% 31% 9% 13% 3362 69%

82% 53% 27% 9% 48% 51% 54% 54% 32% 41% 0% 0% 7% 10% 15% 3% 66%( 49%) 0% 0% 0% 25% 9% 15% 3608 0%

82% 55% 31% 13% 50% 52% 54% 56% 38% 41% 0% 0% 8% 11% 16% 4% 73%( 52%) 0% 0% 0% 24% 9% 14% 4441 0%

82% 51% 28% 10% 49% 52% 54% 57% 35% 41% 0% 0% 7% 10% 16% 4% 65%( 48%) 0% 0% 0% 24% 9% 21% 3778 0%

78% 47% 22% 6% 43% 45% 47% 50% 30% 38% 0% 0% 7% 9% 14% 3% 58%( 45%) 0% 0% 0% 20% 8% 14% 3394 0%

90% 74% 55% 31% 66% 68% 71% 70% 55% 44% 0% 0% 9% 16% 14% 7% 110%( 64%) 13% 0% 0% 32% 7% 11% 3786 79%

88% 63% 39% 16% 56% 58% 62% 64% 40% 42% 0% 0% 8% 14% 16% 6% 77%( 52%) 0% 0% 0% 37% 10% 15% 2887 100%

87% 60% 35% 15% 53% 57% 59% 60% 38% 39% 0% 0% 8% 12% 15% 4% 78%( 52%) 0% 0% 0% 34% 9% 13% 3062 100%

87% 61% 36% 15% 55% 58% 59% 61% 41% 47% 0% 0% 7% 9% 15% 5% 81%( 54%) 0% 0% 0% 28% 11% 16% 4186 13%

79% 51% 27% 10% 46% 49% 52% 52% 32% 42% 0% 0% 8% 10% 14% 3% 63%( 48%) 0% 0% 0% 23% 7% 14% 3755 0%

78% 47% 23% 8% 43% 47% 47% 49% 30% 40% 0% 0% 7% 9% 13% 4% 58%( 44%) 0% 0% 0% 19% 8% 14% 3670 0%

79% 45% 20% 5% 43% 46% 48% 50% 28% 39% 0% 0% 7% 9% 14% 3% 54%( 43%) 0% 0% 0% 19% 8% 20% 3212 0%

88% 61% 35% 13% 55% 57% 61% 64% 38% 50% 0% 0% 7% 9% 18% 3% 75%( 54%) 0% 0% 0% 29% 11% 17% 4536 0%

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

88% 66% 41% 18% 58% 61% 64% 65% 43% 46% 0% 0% 10% 14% 15% 5% 82%( 56%) 5% 0% 0% 30% 10% 14% 4534 86%

81% 49% 23% 7% 45% 47% 51% 53% 30% 40% 0% 0% 7% 9% 14% 3% 59%( 46%) 0% 0% 0% 22% 10% 17% 3922 0%

79% 47% 23% 8% 44% 48% 48% 50% 30% 39% 0% 0% 7% 9% 14% 3% 61%( 46%) 0% 0% 0% 20% 8% 14% 4153 0%

81% 50% 25% 8% 46% 50% 50% 51% 32% 39% 0% 0% 9% 11% 14% 4% 63%( 48%) 0% 0% 0% 22% 8% 14% 3717 0%

80% 51% 25% 8% 47% 50% 52% 55% 31% 44% 0% 0% 7% 9% 15% 3% 62%( 47%) 0% 0% 0% 21% 9% 18% 5134 0%

81% 53% 28% 10% 49% 52% 54% 56% 33% 43% 0% 0% 7% 8% 17% 3% 66%( 48%) 0% 0% 0% 25% 8% 18% 3412 0%

87% 63% 38% 16% 56% 59% 62% 61% 42% 41% 0% 0% 10% 14% 15% 5% 79%( 55%) 4% 0% 0% 31% 10% 15% 2404 84%

80% 48% 24% 8% 44% 47% 48% 51% 30% 37% 0% 0% 8% 10% 14% 3% 61%( 45%) 0% 0% 0% 22% 8% 13% 3115 0%

75% 44% 20% 6% 41% 43% 45% 48% 28% 35% 0% 0% 8% 10% 13% 3% 55%( 43%) 0% 0% 0% 20% 8% 12% 2530 0%

84% 56% 29% 10% 50% 52% 54% 55% 37% 45% 0% 0% 9% 10% 14% 5% 71%( 51%) 0% 0% 0% 21% 8% 15% 5269 0%

79% 49% 24% 8% 44% 47% 49% 50% 33% 41% 0% 0% 7% 9% 14% 5% 63%( 47%) 0% 0% 0% 19% 7% 14% 4368 0%

81% 52% 28% 11% 49% 51% 54% 56% 35% 40% 0% 0% 8% 11% 15% 5% 64%( 47%) 5% 0% 0% 21% 10% 18% 2359 13%

79% 49% 26% 9% 45% 48% 50% 51% 31% 36% 0% 0% 8% 12% 14% 4% 61%( 44%) 0% 0% 0% 24% 7% 14% 2984 90%

78% 46% 22% 6% 43% 48% 47% 48% 30% 34% 0% 0% 7% 10% 14% 3% 59%( 44%) 0% 0% 0% 22% 8% 16% 2088 0%

86% 60% 34% 12% 54% 57% 60% 62% 37% 51% 0% 0% 7% 10% 16% 3% 74%( 53%) 0% 0% 0% 26% 10% 17% 6267 0%

85% 58% 32% 12% 53% 56% 58% 59% 36% 51% 0% 0% 7% 9% 15% 4% 73%( 51%) 0% 0% 0% 24% 10% 17% 6826 0%

Thanks,

Larry

radek_kubka · ‎2014-01-04

CPU seems to be quite heavily hammered during tests - it definitely doesn't help with overall performance.

I appreciate this statement not necessarily brings us any closer to a solution.

aborzenkov · ‎2014-01-05

Then trying to tranfer 16MB chunks we get about 380MB/s transfer rates. 22-23 iops. I would of expected a bit more.

Hmm ... your stats show that real IO size is much less, sounds more like 256KB. Probably it gets split at OS level. This could have impact on throughput. May be it is limited by vmfs block size?

atleast 600MB writing to the rg.

But your tests are pure read, not write?

larryblanco · ‎2014-01-06

Then trying to tranfer 16MB chunks we get about 380MB/s transfer rates. 22-23 iops. I would of expected a bit more.

Hmm ... your stats show that real IO size is much less, sounds more like 256KB. Probably it gets split at OS level. This could have impact on throughput. May be it is limited by vmfs block size?

atleast 600MB writing to the rg.

But your tests are pure read, not write?

Not using VMFS. The OS is Windows 2012 on NFS. I have the MS initiator going inside of the VM. It's an iSCSI lun.

I have not posted any write results, as my read result are pretty poor.

Here are the stats with writes:

Load output IO Meter: Transfer Request Size 4K - 100% Random Access - 100% Access Specification - 50% Read/Write Distribution - 4K Aligned IO

sysstat -M 2

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

99% 97% 93% 72% 92% 95% 96% 96% 83% 46% 0% 0% 17% 25% 33% 8% 180%( 86%) 14% 0% 0% 32% 5% 7% 34423 78%

100% 97% 93% 72% 93% 94% 96% 97% 82% 50% 0% 0% 16% 20% 38% 4% 191%( 89%) 10% 0% 0% 26% 7% 7% 38359 24%

100% 99% 94% 70% 93% 96% 96% 98% 84% 51% 0% 0% 17% 21% 39% 7% 187%( 87%) 2% 0% 0% 32% 7% 9% 38974 73%

100% 97% 92% 71% 92% 95% 96% 95% 81% 48% 0% 0% 17% 24% 34% 6% 179%( 87%) 14% 0% 0% 31% 7% 7% 34195 61%

99% 98% 94% 74% 93% 96% 97% 97% 83% 58% 0% 0% 15% 16% 40% 4% 195%( 89%) 0% 0% 0% 31% 6% 7% 39043 22%

95% 89% 82% 64% 85% 86% 89% 88% 77% 42% 0% 0% 14% 21% 30% 8% 166%( 80%) 13% 0% 0% 34% 5% 7% 28180 85%

100% 97% 93% 72% 93% 94% 96% 98% 82% 49% 0% 0% 17% 20% 39% 5% 192%( 89%) 7% 0% 0% 28% 7% 7% 38026 82%

100% 99% 93% 72% 93% 96% 96% 98% 83% 46% 0% 0% 17% 24% 38% 6% 186%( 89%) 8% 0% 0% 34% 7% 7% 36300 100%

99% 97% 93% 72% 92% 94% 96% 96% 82% 46% 0% 0% 18% 24% 38% 6% 185%( 87%) 13% 0% 0% 28% 5% 7% 36161 63%

100% 97% 93% 70% 93% 96% 96% 96% 82% 51% 0% 0% 17% 18% 43% 5% 194%( 88%) 0% 0% 0% 27% 7% 9% 40822 47%

100% 97% 92% 72% 92% 94% 96% 97% 82% 44% 0% 0% 16% 23% 34% 6% 183%( 88%) 14% 0% 0% 35% 7% 7% 31835 96%

96% 89% 81% 61% 84% 86% 88% 87% 75% 43% 0% 0% 13% 17% 29% 6% 168%( 81%) 12% 0% 0% 32% 7% 8% 26509 62%

100% 98% 93% 72% 93% 96% 96% 97% 82% 51% 0% 0% 17% 20% 39% 6% 188%( 87%) 1% 0% 0% 34% 7% 7% 37206 100%

97% 90% 83% 61% 85% 88% 89% 88% 75% 38% 0% 0% 16% 25% 31% 7% 162%( 81%) 14% 0% 0% 31% 7% 9% 29662 78%

100% 97% 92% 70% 92% 94% 96% 97% 82% 49% 0% 0% 18% 20% 44% 4% 194%( 89%) 4% 0% 0% 18% 7% 9% 43704 30%

95% 89% 82% 61% 84% 86% 87% 88% 77% 42% 0% 0% 16% 22% 32% 8% 165%( 80%) 9% 0% 0% 29% 5% 9% 31094 70%

100% 97% 93% 74% 93% 96% 96% 96% 84% 49% 0% 0% 16% 22% 39% 6% 194%( 90%) 13% 0% 0% 20% 7% 7% 39797 46%

100% 97% 91% 68% 92% 95% 96% 96% 80% 52% 0% 0% 18% 20% 44% 5% 186%( 86%) 0% 0% 0% 25% 7% 10% 42503 76%

99% 96% 90% 67% 91% 94% 95% 94% 82% 43% 0% 0% 15% 23% 29% 7% 172%( 86%) 14% 0% 0% 43% 7% 12% 27077 93%

Load output IO Meter: Transfer Request Size 4K - 100% Random Access - 100% Access Specification - 100% Write Distribution - 4K Aligned IO

sysstat -M 2

ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

100% 98% 93% 73% 93% 95% 96% 96% 86% 52% 0% 0% 8% 16% 32% 7% 197%( 89%) 13% 0% 0% 30% 7% 10% 33940 86%

100% 98% 93% 73% 94% 96% 96% 97% 85% 52% 0% 0% 7% 15% 32% 7% 195%( 89%) 12% 0% 0% 37% 7% 8% 30657 73%

100% 96% 92% 72% 92% 92% 95% 96% 86% 51% 0% 0% 7% 13% 33% 7% 202%( 91%) 12% 0% 0% 29% 7% 8% 34257 68%

99% 97% 92% 69% 93% 93% 95% 96% 88% 49% 0% 0% 7% 14% 32% 8% 200%( 90%) 13% 0% 0% 29% 6% 14% 33422 72%

100% 96% 91% 70% 92% 92% 94% 96% 87% 47% 0% 0% 7% 17% 32% 8% 195%( 90%) 17% 0% 0% 28% 7% 11% 33501 71%

98% 91% 82% 58% 86% 86% 86% 89% 81% 48% 0% 0% 7% 13% 28% 10% 177%( 83%) 11% 0% 0% 27% 8% 12% 29985 63%

100% 99% 95% 74% 94% 95% 96% 97% 88% 48% 0% 0% 8% 18% 30% 8% 198%( 90%) 19% 0% 0% 33% 7% 7% 30909 82%

100% 97% 92% 72% 93% 93% 95% 96% 86% 52% 0% 0% 7% 15% 32% 7% 196%( 89%) 12% 0% 0% 34% 6% 8% 31736 79%

99% 96% 90% 69% 92% 94% 95% 95% 84% 50% 0% 0% 7% 15% 31% 7% 190%( 88%) 13% 0% 0% 34% 7% 11% 30911 74%

100% 97% 92% 70% 93% 94% 96% 97% 85% 54% 0% 0% 6% 12% 36% 6% 203%( 91%) 11% 0% 0% 23% 7% 13% 38043 68%

99% 97% 91% 69% 92% 94% 95% 95% 85% 54% 0% 0% 7% 13% 34% 7% 195%( 88%) 11% 0% 0% 28% 7% 13% 35664 74%

98% 97% 91% 69% 92% 93% 95% 95% 85% 55% 0% 0% 7% 14% 34% 7% 193%( 88%) 15% 0% 0% 24% 7% 12% 35473 64%

100% 99% 94% 74% 94% 95% 96% 97% 88% 49% 0% 0% 8% 20% 30% 7% 195%( 91%) 22% 0% 0% 30% 5% 8% 32008 74%

100% 98% 93% 72% 93% 94% 96% 96% 87% 54% 0% 0% 7% 15% 35% 7% 199%( 89%) 13% 0% 0% 26% 6% 9% 36026 79%

100% 97% 92% 72% 93% 95% 95% 96% 86% 53% 0% 0% 8% 15% 33% 9% 195%( 87%) 12% 0% 0% 31% 8% 10% 33159 81%

100% 97% 92% 71% 93% 94% 96% 98% 84% 59% 0% 0% 6% 13% 33% 7% 191%( 89%) 12% 0% 0% 32% 8% 10% 31931 67%

99% 97% 92% 72% 93% 94% 96% 96% 85% 57% 0% 0% 7% 14% 33% 7% 196%( 89%) 12% 0% 0% 29% 6% 9% 33729 74%

96% 90% 82% 62% 86% 86% 88% 89% 79% 55% 0% 0% 7% 13% 27% 8% 178%( 83%) 12% 0% 0% 26% 6% 10% 31456 64%

97% 90% 83% 63% 86% 87% 90% 91% 77% 57% 0% 0% 7% 13% 28% 7% 173%( 81%) 13% 0% 1% 29% 8% 10% 30854 71%

rmatsumoto · ‎2014-01-06

That bottom sysstat -M 2 output and some of the output in the middle suggest to me that the task is CPU-bound, which is consistent with how our FAS32xx perform in our environment. When you say that the aggr is a 23-disk RG, is it a regular aggr with nothing but SSDs, or is it a flashpool aggr? Either way, the delay is somewhere beyond your disks, based on what you posted. Right now I'd lean towards the head itself(at least for some of those tests).

larryblanco · ‎2014-01-07

NetApp said the same thing. I was on the phone with them with one of their performance team support personnel. He said what we are getting will be pretty much it. There is a mismatch between the SSD and the filer head. The SSD's can outperform the head 3-5:1.

It is a 23 disk RG running RaidDP. We are not using a flashpool.

I really wish that we were told ahead of time that hey, yes you are going to get better performance iops wise compared to the sas disks you have now but you won't be able to get the full potential of the disks with your current setup. Your 3240 will max out at about 38K iops. A 3250 will max out at about 50K.

Larry

rmatsumoto · ‎2014-01-07

No doubt(on the disappointment, given their cost)... I would consider splitting up the SSDs to flashpool aggrs to increase the chance of realizing the performance benefit of SSDs on multiple aggregates. If you have multiple flashpool aggrs, you'll have multiple aggrs that can push the max iops your controllers will allow. They won't be able to do them all at the same time, but at least you won't have one fast SSD-only aggr and slower aggr(s) that are HDD-only, provided that you have HDD-only aggrs on these heads. We've done our own tests with SSDs and have been satisfied with what a flashpool aggr can do, as opposed to SSD-only aggr, especially when we take the price/performance/space ratio into account. You'll burn the first 2 SSDs disks in a flashpool aggr to parity so you won't be able to make too many flashpool aggrs(and tying up these SSDs to parity disks make you cringe, doesn't it?), but that might be more palatable than having one 23-disk SSD aggr that's capable of way more iops than the controller. .

radek_kubka · ‎2014-01-14

Just be wary of the FAS3240HA limit re Flash Pool (assuming no Flash Cache present) - max 0.6TB per controller / 1.2TB per HA pair.

thomas_glodde · ‎2014-01-14

Saw a few FAS3240 with non-ssd setups, they deliver about 400mb write and 1000mb read with a few sas disks.

Properly setting up iSCSI, tuning frame size as well as tcp chimney within windows might be an issue. I´d suggest to run multiple vms with multiple luns and see if performance at least rises when running 2, 3 or 4 VMs with the same tests. But expect the FAS3240 to be CPU bound with small random i/o as kahuna can only split over 2 cores, reaching 200% max.