Hello Carlos,
Those are useful screenshots! The first shows key info which is that the workload is IOPS-dominated: low MB/s values. Similar note on IOPS vs. bandwidth:
https://community.netapp.com/t5/ONTAP-Hardware/IOPS-for-SATA-SAS-FC-disks/m-p/51499/highlight/true#M3641
A "back of the envelope" type of calculation might say 24 x 120 IOPS = 2880 IOPS and yet you're seeing peaks of "only" 1000-1500. But as the link I just provided mentions, that (simply summing up IOPS of all the spindles) wouldn't be the right away to think about it in the context of RAID devices, and then also of storage arrays.
The bulk of it stems from the way RAID works. It seems you have DDP pools which are like RAID 6.
RAID 5 and RAID 6 have an overwrite penalty- that is when you want to modify a single block on a RAID stripe, you need to read it first (e.g. from 10 disks, in RAID 6 made of 8D+2P), and then write it back and although the client modified just one small record, to commit back the stripe changes and results in writes to 10 disks.
Usually there's more writes that can be combined and there are various optimizations that RAID controllers implement, but in any case these are known facts and random writes consume disproportionately more resources, which is why for databases and random IO in general you'll see RAID5 more often recommended (lower, but still significant, write penalty).
In the case you wonder "why do I have DDP if RAID5 is better", the answer is RAID 5 isn't better, it's just different and there are reasons why DDP is more suitable for your use case and configuration (the main one being with 24 disks you can't just put all of them in a RAID 5 group, and optimal RAID 5 group size is 8+1 which means you'd end up with something worse if you used 24 disks here).
So, long story short, all this is as expected.
Additionally, notice how the first chart shows IOPS for a 7d period but MB/s for a 5 min period?
The IOPS are averaged, which means short spikes are probably higher than 1500, but you no longer can see them. If you kept the UI open with a 5 min moving window, you'd probably notice that short term spikes are actually higher than they appear when down-sampled after several days.
What can you do now? Well, you can't get more IOPS than there are available - if we could do it we'd have done it by default.
Less destructive option: VMware (depends on version) has the ability to limit IOPS on VMs. You can identify the VMs with high IOPS (high bandwidth VMs can probably can be ignored, unless they are the same as high IOPS VMs) and do something to constrain their IOPS. This is low risk and done outside of E-Series, so maybe you can check with your reseller or try on your own. As far as I know because it's a client-side thing, NetApp doesn't have its own recommendations do this. Of course, the end result is the constrained VM will be slower and less responsive (or even stall, if constrained too much), and outcomes depend not only on VMware version, but also the guest OS and application, so I'd lower those IOPS few percent at a time (from the maximum observed) and observe for a week. Lowering the number of CPUs or RAM on selected guests is another approach, and similarly in the app (e.g. limit RAM available to SQL Server, etc.). All impact application responsiveness and performance, as expected. Other options include general VMware best practices for guest OS - maybe disabling antivirus scanners or excluding some directories from scanning, etc.
More destructive option: backup and restore fest. To change data protection from DDP to RAID 5 (for example), you would have to backup VMs, maybe try to restore several to see it works, destroy VMware datastores, remove datastores from VMware, destroy E-Series volumes, destroy DDP, create new RAID config and then volumes and VMware datastores, followed by a VM restore.
What would be a "faster" configuration? RAID 10 is fastest, but you'd lose a lot of capacity, so probably not an option. RAID 5 is next option that can help you squeeze a bit more out of those disks (maybe 10- 20%? Is it worth the trouble and risk? I don't know). As indicated earlier you won't do better if you don't create optimal RAID 5 groups, so what you could do is 2 groups of 8+1, and the rest (24-19=5) would probably have to be 1 global hot spare (assuming your SANtricity is not ancient), and then you have 4 disks left, maybe do a RAID 10 from it?
So with that you'd get 3 little "RAID islands" (which is why most customers prefer one DDP island), and lose quite a bit of capacity, and now you need to know which VMs need to go where, because unlike with the current DDP setup where any VM on any datastore gets IOPS from all 24 disks, here it can't go beyond the RAID group so you need to be very aware of what each VM does when you "micro-manage" RAID groups otherwise you may need to do frequent Storage vMotion-ing which can create a lot of workload of its own...This is why DDP is an easy and best overall option in cases like yours - no big downsides, just a small-to-moderate sacrifice in IOPS.
But, assuming you have detailed insights into what each VM does, you could put the busiest VM(s) on RAID 10 and the rest on volumes on the two RAID 5 groups.
It's hard to say how much this would help, but at least it would insulate busy from not busy VMs. The busy VM on a DS on a volume on the RAID 10 group would make only those disks busy, while the rest would be responsive.
Considering the complexity (need to analyze and monitor performance, maybe do Storage vMotion exercises over weekends?) and potential risks deleting and restoring VMs, I'd probably try the less destructive approach first. This approach is time-consuming and finding a better layout than DDP may cost you more time (and trouble) than SSDs, so normally people use it for larger databases (where there's just 1 thing going on) and put transaction logs on RAID 10, tables on RAID 5, etc. For VMs, where there's a bunch of them and maybe even churn, it's hard and you may have all VMs doing lots of IOPS at different times...
