Subscribe

Strange loss of iops

Hello, community!

 

It's time to start my first thread. 

 

Two days ago i started to recive messages from my OnCommand Performance Manager "vol_DS_1 is slow at the data processing node". When i looked at the graph, i had seen that my IPOS moved from ~2000 to ~200. 

As you can see from graph momently increased latency, a lost iops.

 

fc77c059ced80cf7fe0eb60b59a57d58.jpg

First of all i checked my system and i can't find any issues, problems or error messages. 

I called to system administrator, which uses systems located at this NetApp, but he say, that haven't seen any performance issues and all systems works fine, also they don't perform any works or modifications. 

 

My system:

NetApp FAS 2552 with ONTAP 8.3.1. (SAS 10k HDD, no Flash Pool or Flash Cache)

Connection method iSCSI, using 2x10g from each node. 

4 Volumes (DS_1, DS_2, DS_3. DS_4), 4 LUNs, no snapshots enabled. 

 

Clients:

5xVMware vSphere 5.5 Hypervisors (2x10GB NIC) with 10-15 vms and one Oracle RAC.

All VMs (exclude RAC DB disks) located at the problem volume. Also i use 3 another volumes from this NetApp, one for DB files, two for fileserver.

From vSphere i can see avg latency for vol_DS_1 5-20ms, for other 3 volumes ~1ms. 

 

Thanks in advice!