High diska activity

[rgraves2572]

I'm having issues with a FAS 2050 with near 100% disk utilization but can not track down the cause.

We have two aggregates:

aggr0 > 10 disk 1TB Sata

aggr1 > 13 disk 1TB Sata

aggr0 is the snapmirror destenitation for multiple volumes

aggr1 is a snapvault destenitation for SQL / Exchange volumes

IOPS are around 23

Inbound SnapMirror traffic is about 3 to 6mbs

CPU 35%

Disk utilization remains near 100% even when there is no snapmirror traffic. There appears to be a consistent 10000 kB/s read occuring on the disks.

-Robert

[reinoud7]

Hi Robert,

This is an intersting puzzle.

I would do the following things:

• Have you a hot disk (1 disk at 100%) or are all the disks buzy: try statit -b (wait a minute) and than you do statis -e. You get a lot of information but the interesting part is: Disk Statistics. You can see all your hotdisks.
• Are your aggregates almost full?
• When you stop all the snapmirror and snapvault: you see still this high disk utilization? No client access or ndmp? If this is the case, maybe there is a wafle process running (dedup, reallocation, snapshot cleaning, ...).

Hopes this help a little bit.

Reinoud

[rgraves2572]

statit shows that all disks in aggr0 are running 95 to 100% even when there is no snapmirror or snapvault traffic. Parity disks are at like 4%

Dedupe is not running on anything

I checked reallocation is turned on but not sure if there are any jobs, not sure how to check that

Not sure how to check snapshot cleaning.

-Robert

[reinoud7]

Reallocation can you check very easy: first, you find in the message files normaly the start and the end of the reallocation. You can also use wafl scan status to see the background wafl scans. It's normal that you see a lot there, but reallocation is also mentioned.

Reinoud

[rgraves2572]

I did the wafl scan status and there is several things going on

If the system is attempting to reallocate volumes how do you stop it?

These are all snapmirror destentiations so not sure we really want to run reallocate so often

-Robert

[reinoud7]

Hi Robert,

Depending on the ontap version, there is in the more recent versions the reallocate commande (I was forgotten this). Reallocate status gif you the results. With reallocate stop you can stop one process.

I think that a reallocation on a snapmirror destination is not very useful. I do this never, and I think that this maybe more problems than it solves. I would turn this off.

Reinoud