Deletion of files over NFS slow

bader · ‎2009-09-02

I have a FAS 3040 with PAM card, and I have an NFS volume where my Data base admin dumps data. He then runs some jobs on this data and then deletes it. He does this for various data bases in sequence.

We are currently getting volume full errors. He is thnking that after he deletes the data, the filer takes a while to actually physicall delete the data, so he gets volume full when he tries to copy more data to the volume. Does any one know a way to verify this, or see logs of blockes being deleted on the filer? How fast can a filer delete data?

Thanks

Bob

eric_barlier · ‎2009-09-02

Hi Bob,

the controller is not likely to be the issue here, but lets investigate. Can you explain if this is a LUN (I assume its not) and if there

are snapshots involved? Please run the following commands on the controller:

snap list volume_name

df -h volume_name

vol status -v volume_name

Also, please look into what /etc/messages file is giving at the time of the error.

Eric

bader · ‎2009-09-03

Eric,

Thanks for the reply. This is not on a lun, it is an NFS volume and there are no snapshots. The sys log shows no errors. You can actually see what happens buy.

Setup a volume and publish via nfs.

from the host (I used a unix host) mount the volume.

I then created a 10 GB file on the volume (mkfile 10Gb test)

I then do DF -h and look at the space usage on the volume.

I then delete (using rm) the 10 GB file

when you run the rm command you get your prompt back and can do other things

However, I then run DF -h (over and over again) and you can see the space freeing up on the volume as that file is deleted in the background. It takes about 40 seconds to delete the 10 Gb file.

so if you have a 500 GB you are deleting it can take around 33 minutes to delete 500 GB.

I also talked to NetApp and they say there is no ay to adjust this, I have asked them if flexscale priority would make any difference.

Bob

eric_barlier · ‎2009-09-03

hi Bob,

I reckon that wafl freeing up blocks in the background. Flexscale is to ensure writes are done in a certain order so my guess is that

it wont help, but maybe it will. Also flexscale is only useful if you have mixed disk types on any 1 controller.

Does this slow deletion prevent any new writes? I wouldnt think so. Have you tried deleting that 10GB file and then write another one whilst

its deleting? I m not sure IF you can test this but that would be interesting to do.

you could also compare your NFS deletion with a filer local deletion as all the command to run the test you did are on the controllers

mkfile (in priv set below priv set advanced)

rm

df

None of the above will provide a fix but maybe more clarity for starters.

Cheers,

Eric

bader · ‎2009-09-04

Eric,

Thanks,

While the files are being deleted you can't use that space, that's the issue, the script detelets fiels, then tries writtting more data but the space hasn't ckeard yet for the new data. I opend a call with NetaApp and was tolf

"ONTAP is already as optimized as possible for deleting files. So much so that we have a bug where, when there are a very large number of files or very large files being deleted, it takes up all the processing time and other processes are put on hold until the deletes finish, causing the filer to appear unresponsive. See http://now.netapp.com/NOW/products/csb/csb0803-03.shtml

So it looks like this is just the way it is. I will look at creating a sperated volume or adjusting the scripts.

Bob