Files disappear from NetApp 2220 every day at 00:00

[Antancela]

For 5 days at exactly 00:00 files and folders start to disappear from our NetApp , the process continues for exactly 40 minutes. Last night we disconnected it from LAN, but it happened again, so we assume there must be something wrong with NetApp itself. Each time it deletes up to 3 TB of data (yes, that's right). According to diagnostics in UI everything is fine.

Did anyone experienced anything like that, or do have any ideas what could it be and how can we stop it?

[Ontapforrum]

Hi,

Interesting!

FILER will not delete anything automatically, unless it is asked to do so either by user action or by enabling certain feature.

Could you give us more info on :

1) How did you discover the files/folder deleted: Has the volume space decreased ? or the data missing from CIFS/NFS shares?
2) What ONTAP version you are running ? and :
a) Has anyone changed the snapshot schedule policy ? For example : Has anyone reduced the number of 'nightly' snaps?
b) Is 'volume snapshot autodelete' feature enabled on volumes ?

Thanks!

[Antancela]

Hi, 

1) We found out that free space on the volume is growing and then that certain folders are become missing.

2)  SW VERSION: 8.1.1RC1 7-MODE (FAS2240A-SSA-R5)

a) No, nobody changed any settings at all.

b) Could you please tell how to check it.

[Ontapforrum]

Please give us this info for the volume concerend (where free space is growing)

filer> snap autodelete <vol_name>  show
filer> snap list <vol_name>
filer> snap sched <vol_name>

[Antancela]

snap autodelete data_vol show

snapshot autodelete settings for data_vol:

state                : off

commitment            : try

trigger                : volume

target_free_space        : 20%

delete_order            : oldest_first

defer_delete            : user_created

prefix                : (not specified)

destroy_list            : none

snap list data_vol

Volume data_vol

working...

   %/used       %/total  date          name

----------  ----------  ------------  --------

   0% ( 0%)    0% ( 0%)  Dec 19 16:00  hourly.0

   0% ( 0%)    0% ( 0%)  Dec 19 00:00  nightly.0

   0% ( 0%)    0% ( 0%)  Dec 18 20:00  hourly.1

   0% ( 0%)    0% ( 0%)  Dec 18 16:00  hourly.2

   0% ( 0%)    0% ( 0%)  Dec 18 12:00  hourly.3

   0% ( 0%)    0% ( 0%)  Dec 18 08:00  hourly.4

   0% ( 0%)    0% ( 0%)  Dec 18 00:00  nightly.1

   0% ( 0%)    0% ( 0%)  Dec 17 20:00  hourly.5

  snap sched data_vol

Volume data_vol: 0 2 6@8,12,16,20

[Ontapforrum]

Thanks. Could you also attach:

1. /etc/log/messages

2. Autosupoort : atleast 2 weeks old.

[Antancela]

Please find messages attached.

Autosupport folder has over 3000 files. I can see that some files contain information, that I'm not allowed to make public.

Could you please specify certain file names that can be useful.

[Ontapforrum]

From messages log: [I see the culprit in the messages log here, especially due to timing @ 00 hrs]
Line 249: Mon Dec 16 00:20:52 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.2' in aggregate 'aggr0' to recover storage
Line 250: Mon Dec 16 00:20:56 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.1' in aggregate 'aggr0' to recover storage
Line 251: Mon Dec 16 00:20:57 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 252: Mon Dec 16 00:21:01 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'nightly.0' in aggregate 'aggr0' to recover storage
Line 494: Tue Dec 17 00:19:20 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 1251: Wed Dec 18 00:53:01 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 1252: Wed Dec 18 00:53:04 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'nightly.0' in aggregate 'aggr0' to recover storage
Line 2296: Thu Dec 19 00:56:31 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.2' in aggregate 'aggr0' to recover storage
Line 2297: Thu Dec 19 00:56:35 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.1' in aggregate 'aggr0' to recover storage
Line 2298: Thu Dec 19 00:56:38 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'hourly.0' in aggregate 'aggr0' to recover storage
Line 2299: Thu Dec 19 00:56:40 MSK [phobos:wafl.snap.autoDelete:info]: Deleting snapshot 'nightly.0' in aggregate 'aggr0' to recover storage

Thu Dec 19 14:00:43 MSK [phobos:wafl.snap.autoDelete.deleteStateSnap:info]: Deleting snapshot 'autodelete_base' in aggregate 'aggr0' as a new scheduled snapshot has been created.
Thu Dec 19 14:00:43 MSK [phobos:wafl.snap.autoDelete.deleteStateSnap:info]: Deleting snapshot 'autodelete_base' in aggregate 'aggr0' as a new scheduled snapshot has been created.
Thu Dec 19 14:00:43 MSK [phobos:wafl.snap.autoDelete.deleteStateSnap:info]: Deleting snapshot 'autodelete_base' in aggregate 'aggr0' as a new scheduled snapshot has been created.

Fri Dec 20 09:00:43 GMT [phobos:wafl.vol.guarantee.fail:error]: Space for volume data_vol is NOT guaranteed

Question : Is data_vol on 'aggr0' ?

Some explanation about autodelete: (Interesting stuff..hmm)
If aggregate snapshots are enabled and there is not adequate space reserved for these snapshots, a situation may develop where a very large aggregate snapshot is created. This may occur if large volumes are deleted.If SyncMirror triggers an aggregate level snapshot and there is not enough space for two snapshots, the existing snapshot is deleted by the autodeletion feature and instantly replaced by the autodelete_base snapshot.

https://kb.netapp.com/app/answers/answer_view/a_id/1043638

Recommendation in general: {Disable snapshot on aggr}
Disable automatic aggregate Snapshot copy creation by entering the following command:
aggr options aggr_name nosnap on
aggr_name is the name of the aggregate for which you want to disable automatic Snapshot copy creation.

Delete all Snapshot copies in the aggregate by entering the following command:
snap delete -A -a aggr_name

Set the aggregate Snapshot reserve to 0 percent by entering the following command:
snap reserve -A aggr_name 0

[mjdalton1]

I also highly recommend you get off the "Release Candidate"!  It is pre-GA

 SW VERSION: 8.1.1RC1 7-MODE (FAS2240A-SSA-R5)

[Antancela]

mjdalton1, thank you, we'll update it ASAP.

[mjdalton1]

@Antancela I'm going to assume you do not have support. If that's the case only upgrade to 8.1.4P1 (licenses change @ 8.2)   https://mysupport.netapp.com/NOW/download/software/ontap/8.1.4P1/

If you do have support upgrade to 8.2.5 (Be sure to read the Upgrade notices and KB articles)

https://mysupport.netapp.com/NOW/download/software/ontap/8.2.5/download.shtml

[paul_stejskal]

If files are getting deleted, you need to monitor where it is coming from. Fpolicy might be one option, or setting up a packet trace for when it happens might be another option.

https://kb.netapp.com/app/answers/answer_view/a_id/1029832

In there, you would be looking for SMB/SMB2 remove type calls.

ONTAP will not delete data by itself as that would compromise the very thing ONTAP is designed to do.

[Antancela]

Thank you, we updated to  Release 8.2.5P2 7-Mode: Wed Oct 24 05:14:37 PDT 2018.