topic Re: vol0 with many write IOPS in VMware Solutions Discussions

vol0 with many write IOPS

alexander_bahia — Thu, 05 Jun 2025 06:43:32 GMT

Hi,

Somebody knows what causes the many write IOPS on volume vol0 ?

I see ~1500 - 2000 IOPS on volume vol0 interleaved by one hour every day.

There is none snap scheduled at this time, nor snapmirror.

Storage: FAS3170

Ontap: 7.3.1.1

vol0 with many write IOPS

pascalduk — Tue, 18 Oct 2011 06:04:54 GMT

I assume that the peaks are write ops. If that is the case then I think this could be caused by ontap writing the hourly performance statistics to the /etc/log/cm_stats_hourly file. I see an hourly write peak on all the root volumes.

Still, 1500 ops is huge for just writing performance statistics data, but it is the only hourly action I can think of.

vol0 with many write IOPS

alexander_bahia — Tue, 18 Oct 2011 13:29:06 GMT

Thanks Dukers,

I thought like that, too. But, I did not imagined so many IOPS on a single file 16MB size.

Sometimes, these peaks are doing high latency in all volumes on storage. So, I'm trying to move some LUNs to another controller.

regards,

Re: vol0 with many write IOPS

stevedegroat — Thu, 03 May 2012 01:03:33 GMT

I currently have an open case on this. My 6240 controllers are seeing higher Read/Write/Other operations than the production VMware and Oracle volumes. We're looking at the ACP options and possibly disabling that.

Re: vol0 with many write IOPS

colin_graham — Wed, 09 May 2012 09:28:21 GMT

Steve.

Have you had any success with this? - we are seeing the same issue on a 6210 cluster - vol0 has upwards of 1000 iops at times and is consistently either the top vol, or in the top5. Considering our vol0's are alone on a 3 disk aggr. this isnt possible.

Re: vol0 with many write IOPS

stevedegroat — Wed, 09 May 2012 10:13:26 GMT

My open case has resulted in us checking into turning off the ACP (options acp.enabled off). I am going to do that later today on a 6280 cluster hosting my TSM db/logs/storage pools. I'll watch the IOPs via Performance Adviser and post my findings here.

My /vol0 is also just three disks.

Re: vol0 with many write IOPS

stevedegroat — Wed, 09 May 2012 11:32:09 GMT

After disabling ACP on a 6280 /vol0

Re: vol0 with many write IOPS

colin_graham — Wed, 09 May 2012 14:56:18 GMT

Cheers for posting.

Are Netapp saying its a bug? - are you going to leave it disabled?

Thanks

Col G

Re: vol0 with many write IOPS

stevedegroat — Wed, 09 May 2012 15:03:04 GMT

Unknown at this time. Support hasn't said it is a Bug; disabling ACP was a test to see if the IO would in fact drop. Technically there shouldn't be any impact to leaving it disabled (that I have been able to find so far).

Re: vol0 with many write IOPS

colin_graham — Wed, 09 May 2012 15:10:36 GMT

Ive seen a few other reports of the same issue.

Does it have any further implications rather than just messing up performance graphs?

Personally im not too fussed about disabling it, but im not sure those above would see it the same way unless theres a specific bug attached to it.

Re: vol0 with many write IOPS

stevedegroat — Fri, 11 May 2012 12:12:08 GMT

After setting acp.enabled to "off" on three 6200 clusters I have recorded a significant drop in /vol0 IO across all six controllers. I have also noticed a "leveling" of the overall CPU. So far Support has not found any reason not to re-enable ACP (long term issues, etc.). I have had a 3160 running with no issues for almost 2 years without the ACP connected. I have one more system, primary VMware/NFS/iSCSI 6240 array which promoted my quest for a solution to the high IO, left to do (scheduled for Sunday morning).

Re: vol0 with many write IOPS

colin_graham — Fri, 11 May 2012 12:15:39 GMT

Thanks for the update... ill set the ball rolling on disabling ours then.

Re: vol0 with many write IOPS

ckyle — Fri, 11 May 2012 15:35:45 GMT

NetApp has a bug that might be similar to your issue. Bug ID is 526941, but the url is http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=526941.

Another option that might work is to disable the auto firmware update check. The command to do that is "options acp.fwAutoUpdateEnabled off" This gives you the benefit of having ACP, while potentially fixing this bug.

Re: vol0 with many write IOPS

stevedegroat — Sat, 12 May 2012 11:45:05 GMT

Yep, my Support case just came back with that Bug. I'll try applying that to the machines where I have already disabled ACP, then re-enable ACP to see if it keeps the IO down.

Remove checksum firmware files from "/mroot/etc/acpp_fw". Specifically removing

following files will avoid this issue:

* ACP-IOM3.xxxx.AFW.FVF

* ACP-IOM6.xxxx.AFW.FVF

where; "xxxx" is the firmware file version.