We have a FAS2040 hosting VMware luns and Exchange luns using ISCSI. The environment is rather small and performance is nice overall. However, we've noticed that everyday, the both CPUs of the second controller peak at about 90-95%. This results in a drop of overall IOP/s and affects all the LUNs. This occurs precisely between 13:00 and 13:10. Not before, not after.
No specific jobs are supposed to be running at that time either on the Netapp side, Exchange side or VMware side (including servers). I don't see any tasks that would do this other than one on the NetApp controller.
I've attached some diagnostics command in priv set diag.
sysstat -x 2
sysstat -M 1
and so on.
I've determined that the Kahuna domain was taking about 70-80% of the CPU between 13:00 and 13:10. It normally takes about 10-15%. Now, I don't know where to look to determine what is running behind the Kahuna process.
Your help in pinpointing where is the issue or the task running would be greatly appreciated.
Do you know if there is a way to check if something is scheduled to run at a specific time on ONTAP? Apart from that, we have snapshots running at 12:00PM but they all finish around 12:02-12:05 and they don't cause this CPU peak!
Looks like something is hammering the disks at that time and the system is having problems keeping up and is constantly doing consistancy points. Also the disk utilisation is very high even for a SAS/FC system. Seems strange tho you are only seeing this during that specific time.
As this is a 2000 system have you downloaded DFM and used ops mgr (inc lic with system)? in there is the Performance advisor module which would help with the monitoring of the system.
I looked through the NetApp Management Console and one of the luns would bump the total throughput/sec from ~1 000 000 to ~50 000 000 bytes per seconds. That lun is a VMware lun which is hosting a SQL Server. For some reason, the DBA scheduled transactional logs backup and a database dump at that specific time.
I'll have to talk to the DBA for this. Thanks all for your help, it helped me understand more Netapp controllers and how awesome they are