Refreshing storage installation - NetApp e-2824 - questions about iSCSI performance

CarlosCoque

Hello everyone,

Around 5 years ago, I installed a SAN to serve as a repository for the VMs from 2 VMware servers.

It was my first time working with a SAN, so I had numerous issues (post below).

https://community.netapp.com/t5/EF-E-Series-SANtricity-and-Related-Plug-ins/First-storage-installation-NetApp-e-2824-questions-about-iSCSI-performance/m-p/141702#M619%3Fref_source=cty-other%20campaign%20%7Budf%7D--1957

At the end of the installation, the performance was acceptable, so we decided to stick with that.

We used 1GB/s connections between the SAN (that had 10GB/s NICs), a switch, and the 2 VMware servers.

Our needs increased and we had to keep adding VMs to that system and it started to be slow.

I decided to do an upgrade and added 2 x 10GB/s switches and 10GB/s NICs to the 2 VMware servers, so right now all pieces of equipment in that environment are connected at 10GB/s instead of 1GB/s.

I expected that just doing that, would give me not only HA (I added a separate path to each of those 2 switches) but also an increase in performance, but that hasn't happened and the performance seems kind of the same.

We're currently using active/passive in VMware NICs and DelayedAck is disabled.

Does anyone have any idea on what else I can investigate or if it has any good practice that I could now be aware of?

Thanks,

Carlos

elementx

What seems to be missing from the other post (I skimmed through it) is key detail: E-Series storage configuration.

Presumably you have mostly random access (with VMware clients), so IOPS are important, which also means that if your disks are not SSD then you need a lot of them to get IOPS you need.

For example with 10+ NL-SAS disks in RAID6 you may get only 2-3K 4kB IOPS.

If SANtricity IOPS perf monitor shows 2-3K IOPS, than that's it, you're maxing out IOPS. At the same time, if you look at 10GigE bandwidth utilization you'll probably discover it's low (3000 IOPS x 4kB = 12 MB/s).

It's not possible to add 2 SSDs to a disk group made of non-SSDs, so to increase IOPS you'd have the following choices:

- Add 2 SSDs as new R1 disk group and turn them into read cache (will help with reads, as long as write ratio is low (10-15%). If write % is higher than that, that may not help

- Add 2 SSDs as new R1, create 1 or 2 volumes on this disk group, create new VMware DS, and move busy VMs to these disks

- Add 5 or more SSDs and create a R5 disk group to get more capacity and more performance (same as the bullet above, but you'd get more usable due to a lower R5 overhead compared to R1

- Add more HDDs (I wouldn't recommend this if you have a lot of IOPS, it's cheaper to buy SSDs for IOPS)

You can also monitor the performance from VMware side. We also have a free way to provide detailed metrics using Grafana (https://github.com/netapp/eseries-perf-analyzer/; requires a Linux VM with Docker inside plus some Docker skills).

What else you can do: maybe check https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Systems/E-Series_Storage_Array/Performance_Degradation_with_Data_Assurance_enabled_Volumes_and_iSCSI - not sure if this applies to your array or not. It may help in marginal ways. If you have HDDs and are maxing out IOPS it probably won't help enough.