Ask The Experts

Highlighted

Files disappear from NetApp 2220 every day at 00:00

For 5 days at exactly 00:00 files and folders start to disappear from our NetApp , the process continues for exactly 40 minutes. Last night we disconnected it from LAN, but it happened again, so we assume there must be something wrong with NetApp itself. Each time it deletes up to 3 TB of data (yes, that's right). According to diagnostics in UI everything is fine.

Did anyone experienced anything like that, or do have any ideas what could it be and how can we stop it?

12 REPLIES 12
Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

Hi,

 

Interesting!


FILER will not delete anything automatically, unless it is asked to do so either by user action or by enabling certain feature.

 

Could you give us more info on :

1) How did you discover the files/folder deleted: Has the volume space decreased ? or the data missing from CIFS/NFS shares?
2) What ONTAP version you are running ? and :
a) Has anyone changed the snapshot schedule policy ? For example : Has anyone reduced the number of 'nightly' snaps?
b) Is 'volume snapshot autodelete' feature enabled on volumes ?

 

Thanks!

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

Hi, 

 

1) We found out that free space on the volume is growing and then that certain folders are become missing.

2)  SW VERSION: 8.1.1RC1 7-MODE (FAS2240A-SSA-R5)

a) No, nobody changed any settings at all.

b) Could you please tell how to check it.

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

Please give us this info for the volume concerend (where free space is growing)

 

filer> snap autodelete <vol_name>  show
filer> snap list <vol_name>
filer> snap sched <vol_name>

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

I also highly recommend you get off the "Release Candidate"!  It is pre-GA

 

 SW VERSION: 8.1.1RC1 7-MODE (FAS2240A-SSA-R5)

 

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

snap autodelete data_vol show

snapshot autodelete settings for data_vol:

state                : off

commitment            : try

trigger                : volume

target_free_space        : 20%

delete_order            : oldest_first

defer_delete            : user_created

prefix                : (not specified)

destroy_list            : none

 

 

 

snap list data_vol

Volume data_vol

working...

 

   %/used       %/total  date          name

----------  ----------  ------------  --------

   0% ( 0%)    0% ( 0%)  Dec 19 16:00  hourly.0

   0% ( 0%)    0% ( 0%)  Dec 19 00:00  nightly.0

   0% ( 0%)    0% ( 0%)  Dec 18 20:00  hourly.1

   0% ( 0%)    0% ( 0%)  Dec 18 16:00  hourly.2

   0% ( 0%)    0% ( 0%)  Dec 18 12:00  hourly.3

   0% ( 0%)    0% ( 0%)  Dec 18 08:00  hourly.4

   0% ( 0%)    0% ( 0%)  Dec 18 00:00  nightly.1

   0% ( 0%)    0% ( 0%)  Dec 17 20:00  hourly.5

 

  snap sched data_vol

Volume data_vol: 0 2 6@8,12,16,20

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

mjdalton1, thank you, we'll update it ASAP.

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

Thanks. Could you also attach:

 

1. /etc/log/messages

2. Autosupoort : atleast 2 weeks old.

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

@Antancela I'm going to assume you do not have support. If that's the case only upgrade to 8.1.4P1 (licenses change @ 8.2)   https://mysupport.netapp.com/NOW/download/software/ontap/8.1.4P1/

 

If you do have support upgrade to 8.2.5 (Be sure to read the Upgrade notices and KB articles)

https://mysupport.netapp.com/NOW/download/software/ontap/8.2.5/download.shtml

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

If files are getting deleted, you need to monitor where it is coming from. Fpolicy might be one option, or setting up a packet trace for when it happens might be another option.

 

https://kb.netapp.com/app/answers/answer_view/a_id/1029832

 

In there, you would be looking for SMB/SMB2 remove type calls.

 

ONTAP will not delete data by itself as that would compromise the very thing ONTAP is designed to do.

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

Thank you, we updated to  Release 8.2.5P2 7-Mode: Wed Oct 24 05:14:37 PDT 2018.

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

Please find messages attached.

Autosupport folder has over 3000 files. I can see that some files contain information, that I'm not allowed to make public.

Could you please specify certain file names that can be useful.

Highlighted

Re: Files disappear from NetApp 2220 every day at 00:00

From messages log: [I see the culprit in the messages log here, especially due to timing @ 00 hrs]
Line 249: Mon Dec 16 00:20:52 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.2' in aggregate 'aggr0' to recover storage
Line 250: Mon Dec 16 00:20:56 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.1' in aggregate 'aggr0' to recover storage
Line 251: Mon Dec 16 00:20:57 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 252: Mon Dec 16 00:21:01 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'nightly.0' in aggregate 'aggr0' to recover storage
Line 494: Tue Dec 17 00:19:20 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 1251: Wed Dec 18 00:53:01 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 1252: Wed Dec 18 00:53:04 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'nightly.0' in aggregate 'aggr0' to recover storage
Line 2296: Thu Dec 19 00:56:31 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.2' in aggregate 'aggr0' to recover storage
Line 2297: Thu Dec 19 00:56:35 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.1' in aggregate 'aggr0' to recover storage
Line 2298: Thu Dec 19 00:56:38 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 2299: Thu Dec 19 00:56:40 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'nightly.0' in aggregate 'aggr0' to recover storage


Thu Dec 19 14:00:43 MSK [phobos:wafl.snap.autoDelete.deleteStateSnap:info]: Deleting snapshot 'autodelete_base' in aggregate 'aggr0' as a new scheduled snapshot has been created.
Thu Dec 19 14:00:43 MSK [phobos:wafl.snap.autoDelete.deleteStateSnap:info]: Deleting snapshot 'autodelete_base' in aggregate 'aggr0' as a new scheduled snapshot has been created.
Thu Dec 19 14:00:43 MSK [phobos:wafl.snap.autoDelete.deleteStateSnap:info]: Deleting snapshot 'autodelete_base' in aggregate 'aggr0' as a new scheduled snapshot has been created.


Fri Dec 20 09:00:43 GMT [phobos:wafl.vol.guarantee.fail:error]: Space for volume data_vol is NOT guaranteed

 

Question : Is data_vol on 'aggr0' ?

 

Some explanation about autodelete: (Interesting stuff..hmm)
If aggregate snapshots are enabled and there is not adequate space reserved for these snapshots, a situation may develop where a very large aggregate snapshot is created. This may occur if large volumes are deleted.If SyncMirror triggers an aggregate level snapshot and there is not enough space for two snapshots, the existing snapshot is deleted by the autodeletion feature and instantly replaced by the autodelete_base snapshot.

 

https://kb.netapp.com/app/answers/answer_view/a_id/1043638

 

Recommendation in general: {Disable snapshot on aggr}
Disable automatic aggregate Snapshot copy creation by entering the following command:
aggr options aggr_name nosnap on
aggr_name is the name of the aggregate for which you want to disable automatic Snapshot copy creation.

Delete all Snapshot copies in the aggregate by entering the following command:
snap delete -A -a aggr_name

Set the aggregate Snapshot reserve to 0 percent by entering the following command:
snap reserve -A aggr_name 0

Check out the KB!
NetApp Insights To Action
All Community Forums