Refreshing storage installation - NetApp e-2824 - questions about iSCSI performance

[CarlosCoque]

Hello everyone,

Around 5 years ago, I installed a SAN to serve as a repository for the VMs from 2 VMware servers.

It was my first time working with a SAN, so I had numerous issues (post below).

https://community.netapp.com/t5/EF-E-Series-SANtricity-and-Related-Plug-ins/First-storage-installation-NetApp-e-2824-questions-about-iSCSI-performance...

At the end of the installation, the performance was acceptable, so we decided to stick with that.

We used 1GB/s connections between the SAN (that had 10GB/s NICs), a switch, and the 2 VMware servers.

Our needs increased and we had to keep adding VMs to that system and it started to be slow.

I decided to do an upgrade and added 2 x 10GB/s switches and 10GB/s NICs to the 2 VMware servers, so right now all pieces of equipment in that environment are connected at 10GB/s instead of 1GB/s.

I expected that just doing that, would give me not only HA (I added a separate path to each of those 2 switches) but also an increase in performance, but that hasn't happened and the performance seems kind of the same.

We're currently using active/passive in VMware NICs and DelayedAck is disabled.

Does anyone have any idea on what else I can investigate or if it has any good practice that I could now be aware of?

Thanks,

Carlos

[elementx · Accepted solution]

What seems to be missing from the other post (I skimmed through it) is key detail: E-Series storage configuration.

Presumably you have mostly random access (with VMware clients), so IOPS are important, which also means that if your disks are not SSD then you need a lot of them to get IOPS you need.

For example with 10+ NL-SAS disks in RAID6 you may get only 2-3K 4kB IOPS.

If SANtricity IOPS perf monitor shows 2-3K IOPS, than that's it, you're maxing out IOPS. At the same time, if you look at 10GigE bandwidth utilization you'll probably discover it's low (3000 IOPS x 4kB = 12 MB/s).

It's not possible to add 2 SSDs to a disk group made of non-SSDs, so to increase IOPS you'd have the following choices:

- Add 2 SSDs as new R1 disk group and turn them into read cache (will help with reads, as long as write ratio is low (10-15%). If write % is higher than that, that may not help

- Add 2 SSDs as new R1, create 1 or 2 volumes on this disk group, create new VMware DS, and move busy VMs to these disks

- Add 5 or more SSDs and create a R5 disk group to get more capacity and more performance (same as the bullet above, but you'd get more usable due to a lower R5 overhead compared to R1

- Add more HDDs (I wouldn't recommend this if you have a lot of IOPS, it's cheaper to buy SSDs for IOPS)

You can also monitor the performance from VMware side. We also have a free way to provide detailed metrics using Grafana (https://github.com/netapp/eseries-perf-analyzer/; requires a Linux VM with Docker inside plus some Docker skills).

What else you can do: maybe check https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Systems/E-Series_Storage_Array/Performance_Degradation_with_Data_Assurance_enabled_Volum... - not sure if this applies to your array or not. It may help in marginal ways. If you have HDDs and are maxing out IOPS it probably won't help enough.

[ahmadm]

As workload increases, disks are likely to be the next component to result in slowness. We would need to check some performance data from the system to judge the source of slowness and whether there are any optimizations suitable to the system.

I suggest to open a support case to review the system. If AutoSupport is enabled, the serial number or name of system would help.