topic Re: SnapCreator is not deleting snapshots in Data Protection

SnapCreator is not deleting snapshots

tom_taelman — Fri, 18 Dec 2015 08:27:00 GMT

Hi,

We have upgraded SnapCreator from 3.6 to 4.1.

In 3.6 everything was working fine. We did not change anything to the config.

Now we have running SnapCreator in 4.1 and see that SnapCreator is not deleting snapshots on the primary. There is also no error in the log files.

We have only a comment line in the logfile:

########## Running Snapshot copy Delete on Primary ##########

########## Running Snapshot copy Delete on Secondary ##########

########## Running Snapshot copy Delete on Secondary NETAPP02 ##########

In the configfile, we have the following config:

########################################################################################################################

# Snapshot Options #

########################################################################################################################

SNAME=SERVER15

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RETENTION_AGE=daily:2

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y

NTAP_SNAPSHOT_LABEL=

NTAP_SNAPSHOT_CLEANUP=N

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_RETENTIONS=DAILY:2

NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_DISABLE=N

Is there something changed in the upgrade to 4.1?

Why is SnapCreator not deleting my snapshots on the primary? It was working in 3.6 without any problem

Re: SnapCreator is not deleting snapshots

ktenzer — Mon, 13 Jan 2014 10:05:15 GMT

I believe this is because the policy is case sensitive and in 4.0 the policies changed to upper case.

I see this in config, I would change NTAP_SNAPSHOT_RETENTIONS=DAILY:2 to NTAP_SNAPSHOT_RETENTIONS=daily:2

NTAP_SNAPSHOT_RETENTIONS=DAILY:2

NTAP_SNAPSHOT_RETENTION_AGE=daily:2

Let us know if this works?

Please pay attention to case sensitivity in regards to policies this can be an issue. I think if you use GUI and create policy they will always be upper case. So if you update something in GUI and are using lower case it can put things out of whack. I recommend switching to upper case, of course if you do then you will have to manually remove the lower case policy snapshots (might be too much work).

Regards,

Keith

Re: SnapCreator is not deleting snapshots

tom_taelman — Mon, 13 Jan 2014 10:26:53 GMT

Still the same problem.

See complete config file:

CONFIG_TYPE=STANDARD

VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E

META_DATA_VOLUME=

LOG_NUM=30

SNAPDRIVE=Y

SNAPDRIVE_DISCOVERY=Y

LOG_TRACE_ENABLE=Y

USE_PROXY=N

NTAP_PWD_PROTECTION=Y

NTAP_TIMEOUT=60

USE_GLOBAL_CONFIG=N

FEDERATED_APPLICATIONS=

ALLOW_IP_ADDR=N

SC_TMP_DIR=

########################################################################################################################

# CMode Options #

########################################################################################################################

CMODE_CLUSTER_NAME=

CMODE_CLUSTER_USERS=

CMODE_SNAPSHOT_FORCE_DELETE=Y

CMODE_SET=N

########################################################################################################################

# Snapshot Options #

########################################################################################################################

SNAME=SERVER15

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RETENTION_AGE=DAILY:2

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y

NTAP_SNAPSHOT_LABEL=

NTAP_SNAPSHOT_CLEANUP=N

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_RETENTIONS=DAILY:2

NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_DISABLE=N

########################################################################################################################

# SnapVault Options #

########################################################################################################################

NTAP_SNAPVAULT_UPDATE=Y

NTAP_SNAPVAULT_NODELETE=N

SNAPVAULT_QTREE_INCLUDE=

NTAP_SNAPVAULT_RESTORE_WAIT=Y

NTAP_SNAPVAULT_RETENTION_AGE=daily:7

NTAP_SNAPVAULT_SNAPSHOT=N

NTAP_SNAPVAULT_MAX_TRANSFER=

NTAP_SNAPVAULT_WAIT=360

NTAP_SNAPVAULT_RETENTIONS=DAILY:7

SNAPVAULT_VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E

########################################################################################################################

# SnapMirror Options #

########################################################################################################################

NTAP_SNAPMIRROR_WAIT=60

NTAP_SNAPMIRROR_USE_SNAPSHOT=N

NTAP_SNAPMIRROR_MAX_TRANSFER=

SNAPMIRROR_VOLUMES=

SNAPMIRROR_CASCADING_VOLUMES=

NTAP_SNAPMIRROR_CASCADING_UPDATE=N

NTAP_SNAPMIRROR_UPDATE=N

########################################################################################################################

# Clone Options #

########################################################################################################################

NTAP_CLONE_IGROUP_MAP=

NTAP_NFS_EXPORT_ACCESS=

NTAP_LUN_CLONE_RESERVATION=N

NTAP_CLONE_SECONDARY_VOLUMES=

NTAP_NUM_VOL_CLONES=0

NTAP_NFS_EXPORT_PERSISTENT=N

NTAP_CLONE_FOR_BACKUP=Y

NTAP_CIFS_EXPORT_ENABLE=N

NTAP_CLONE_SECONDARY=N

NTAP_NFS_EXPORT_HOST=

NTAP_VOL_CLONE_RESERVE=NONE

########################################################################################################################

# ProtectionManager Options #

########################################################################################################################

NTAP_DFM_DATA_SET=

NTAP_PM_RUN_BACKUP=N

NTAP_PM_UPDATE=N

NTAP_DFM_SNAPSHOT_FORMAT=

########################################################################################################################

# OSSV Options #

########################################################################################################################

NTAP_OSSV_FS_SNAPSHOT=

NTAP_OSSV_FS_SNAPSHOT_CREATE_CMD01=

NTAP_OSSV_HOMEDIR=

NTAP_OSSV_ENABLE=N

########################################################################################################################

# APP Commands #

########################################################################################################################

UMOUNT_CMD01=

ARCHIVE_CMD01=

APP_UNQUIESCE_CMD01=

MOUNT_CMD01=

APP_QUIESCE_CMD01=

########################################################################################################################

# Pre Commands #

########################################################################################################################

PRE_CLONE_CREATE_CMD01=

PRE_RESTORE_CMD01=

PRE_EXIT_CMD01=

PRE_NTAP_CMD01=

PRE_APP_UNQUIESCE_CMD01=

PRE_APP_QUIESCE_CMD01=

PRE_NTAP_CLONE_DELETE_CMD01=

########################################################################################################################

# Post Commands #

########################################################################################################################

POST_NTAP_DATA_TRANSFER_CMD01=

POST_RESTORE_CMD01=

POST_APP_QUIESCE_CMD01=

POST_NTAP_CMD01=

POST_CLONE_CREATE_CMD01=

POST_APP_UNQUIESCE_CMD01=

########################################################################################################################

# Event Options #

########################################################################################################################

SUCCESS_MSG=INFO: Snap Creator finished successfully ( Action: %ACTION )

NTAP_ASUP_ERROR_ENABLE=N

SENDTRAP=

SUCCESS_TRAP=

########################################################################################################################

# Agent Options #

########################################################################################################################

SC_AGENT_UNQUIESCE_TIMEOUT=3605

SC_AGENT_WATCHDOG_ENABLE=N

SC_AGENT=10.10.21.113:9191

SC_AGENT_LOG_ENABLE=Y

SC_AGENT_TIMEOUT=3600

########################################################################################################################

# Clone Target Options #

########################################################################################################################

SC_CLONE_TARGET=

########################################################################################################################

# Application Options #

########################################################################################################################

APP_IGNORE_ERROR=N

APP_NAME=domino

APP_CONF_PERSISTENCE=Y

APP_DEFINED_RESTORE=N

PYTHON_OPTS=

APP_DEFINED_CLONE=N

JAVA_HOME=

PYTHON_HOME=

FS_NAME=

JVM_ARGS=

PERL_HOME=

PERL_OPTS=

PLUGINS_FORWARD=

VALIDATE_VOLUMES=

APP_AUTO_DISCOVERY=N

JAVA_CLASSPATH=

PLUGINS_RESTORE=

APP_DEFINED_BACKUP=N

APP_DEFINED_UMOUNT=N

PLUGINS_REVERSE=

APP_DEFINED_MOUNT=N

########################################################################################################################

# Archive Options #

########################################################################################################################

ARCHIVE_LOG_RECURSIVE_SEARCH=N

ARCHIVE_LOG_DIR=N:\changeinfo\logs

ARCHIVE_LOG_EXT=TXN

ARCHIVE_LOG_RETENTION=2.0

ARCHIVE_LOG_ENABLE=Y

########################################################################################################################

# Plug-In Parameter #

########################################################################################################################

Notes_ExecDirectory=C:\Lotus\Domino

DOMINO_DATABASE_TYPE=0

DOMINO_DATA_PATH=D:\Lotus\Domino\Data

LOTUS=C:\Lotus

DOMINO_RESTORE_DATA_PATH=D:\Lotus\Domino\Data

DOMINO_CHANGE_INFO_PATH=N:\changeinfo

DOMINO_INI_PATH=C:\Lotus\Domino\notes.ini

I have the same for the SnapVault config and there it is working.

Re: SnapCreator is not deleting snapshots

ktenzer — Mon, 13 Jan 2014 10:31:12 GMT

Yes if the snapshots on controller have lower case daily and you have upper case DAILY in config and it is case sensitive then your existing snapshots will never be deleted.

Options are continue to use lower case or use upper case and rename snapshots on controller.

I believe this is problem based on info but if you could also provide list of snapshots from controller for this config that would help?

Regards,

Keith

Re: SnapCreator is not deleting snapshots

tom_taelman — Mon, 13 Jan 2014 10:39:57 GMT

Here is the list of the snapshots.

The latest created snapshot (after changing the config file to uppencase) is also in lowercase

NETAPP01> snap list SERVER15_D
Volume SERVER15_D
working...
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Jan 13 11:10  SERVER15-daily_20140113110938 (snapvault)
  5% ( 5%)    1% ( 0%)  Jan 12 23:30  SERVER15-daily_20140112233000
 10% ( 5%)    1% ( 1%)  Jan 11 23:30  SERVER15-daily_20140111233000
 14% ( 5%)    2% ( 0%)  Jan 10 23:30  SERVER15-daily_20140110233000
NETAPP01> snap list SERVER15_E
Volume SERVER15_E
working...
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Jan 13 11:10  SERVER15-daily_20140113110938 (snapvault)
  8% ( 8%)    1% ( 1%)  Jan 12 23:30  SERVER15-daily_20140112233000
 17% (10%)    3% ( 2%)  Jan 11 23:30  SERVER15-daily_20140111233000
 24% (10%)    4% ( 2%)  Jan 10 23:30  SERVER15-daily_20140110233000

If it was case sensitive SnapCreator should created the latest snapshot with daily in uppercase?

Re: SnapCreator is not deleting snapshots

ktenzer — Mon, 13 Jan 2014 12:07:13 GMT

Can you change this to uppercase as well?

NTAP_SNAPVAULT_RETENTION_AGE=daily:7

I just ran a 4.1 config with DAILY as uppercase and the snapshots made had daily lowercase and deletion was working. In addition you can create a new config and then compare what differences are, maybe something else is out of whack?

In addition how did you create this config? It looks based on config format you created a new config in the 4.1 GUI? We have tested upgrade and 3.6 configs should run and work unchanged in 4.0 and 4.1.

Regards,

Keith

Re: SnapCreator is not deleting snapshots

tom_taelman — Mon, 13 Jan 2014 13:49:10 GMT

I have the same issue with a complete newly created config via the GUI for another server.

This is the config:

########################################################################################################################

# Snapshot Options #

########################################################################################################################

SNAME=DCSLHOSA01

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RETENTION_AGE=DAILY:5

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

NTAP_SNAPSHOT_LABEL=

NTAP_SNAPSHOT_CLEANUP=N

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_RETENTIONS=DAILY:5

NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD03=sdcli snap create -D F: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_DISABLE=N

In the log files, I see only this:

########## ARCHIVE COMMANDS ##########

[2014-01-13 14:43:00,570] INFO: Archive commands are not defined

########## Running Snapshot copy Delete on Primary ##########

########## Application cleanup ##########

I look like the function to delete snapshots is not working or not started. All other function have a timestamp and at least INFO but the function Running Snapshot copy Delete on Primary is only logged.

Re: SnapCreator is not deleting snapshots

ktenzer — Mon, 13 Jan 2014 13:54:08 GMT

I have passed this on to one of our engineers, he is on EST so should comment on this shortly.

Regards,

Keith

Re: SnapCreator is not deleting snapshots

ktenzer — Mon, 13 Jan 2014 13:57:20 GMT

Oh one more things how are you running the backup jobs? Can you give us example? In addition if you can add a debug log of a backup where something isnt deleted that would be great too.

Keith

Re: SnapCreator is not deleting snapshots

tom_taelman — Mon, 13 Jan 2014 14:06:10 GMT

The backups are started from the scheduler or from the GUI via the drop down menu Actions - Backup and choosing the daily profile

You can find the debug logs of 2 configs with their conf file in attachment

Re: SnapCreator is not deleting snapshots

sivar — Mon, 13 Jan 2014 15:41:44 GMT

Hello Tom,

I just started reviewing DCSLHOSA01 - and hence few questions around that.

1) Do you have snapvault configured for server DCSLHOSA01 volumes?

2) If so, Can you copy paste the snapvault status output from the NetApp controller for the below volumes?

controller: lhoiststo02

Volumes: DCSLHOSA01_F,DCSLHOSA01_E,DCSLHOSA01_D

3) NTAP_CLONE_FOR_BACKUP=Y

Do you really need this parameter? What is the use case?

The reason I am asking is because, I see this in the debug log.

########## Running storage discovery ##########

[2014-01-13 14:42:01,812] INFO: Skipping Snapshot copy discovery

########## Storage discovery finished successfully ! ##########

There is no snapshot discovery and hence (I believe) is no snapshot deletion on the primary.

4) Could you please change this variable to N (if you don't use that cloning feature) and try a backup again?

NTAP_CLONE_FOR_BACKUP=N

NTAP_CLONE_FOR_BACKUP explained:

There may be use cases where a clone needs to be created right after a snapshot is taken.

Example:

snapshot->lun clone->mount->backup to tape->clone delete

snapshot->vol clone->mount->DR test->clone delete

Re: SnapCreator is not deleting snapshots

tom_taelman — Mon, 13 Jan 2014 16:03:08 GMT

The server DCSLHOSA01 has no SnapVault volumes

We did not need the option NTAP_CLONE_FOR_BACKUP. For DCSLHOSA01 the value was already N and for the other server, I have changed it to N

Created a new backup, but still the same issue. No snapshot deleted on primary.

Re: SnapCreator is not deleting snapshots

sivar — Mon, 13 Jan 2014 16:12:01 GMT

Thanks. Let me test this in the lab and get back to you shortly.

Re: SnapCreator is not deleting snapshots

sivar — Mon, 13 Jan 2014 17:14:51 GMT

Are you able to list backups for this profile/config?

snapcreator --server <servername> --port 8443 --user <username> --passwd <password> --profile Oracle --config DCSLHOSA01 --action backupList --policy daily --debug --verbose

Please let me know the output.

Thanks.

Re: SnapCreator is not deleting snapshots

sivar — Mon, 13 Jan 2014 21:13:41 GMT

I am able to reproduce this behavior in the lab.

I am looking into this now.

I will keep you posted soon.

Re: SnapCreator is not deleting snapshots

tom_taelman — Tue, 14 Jan 2014 10:12:11 GMT

I was able to produce the list with the backups for this config without any problem.

See attachment.

Do you have more news for me as you was able to reproduce our problem into the lab?

Re: SnapCreator is not deleting snapshots

sivar — Tue, 14 Jan 2014 14:01:12 GMT

I will file a defect request on this and have our core developers look into it.

Please give me a day or two and I will get back to you on the progress.

Sorry for the inconvenience.

Re: SnapCreator is not deleting snapshots

sivar — Tue, 14 Jan 2014 23:01:55 GMT

Dear Tom,

This issue seems to have been already fixed by the 4.1GA release.

I was able to consistently recreate this issue with 4.1 community release.

With 4.1GA release this works perfectly as expected.

Unfortunately you will have to wait for couple of weeks for the 4.1GA code. (as you can read from this thread https://communities.netapp.com/message/123394#123394 )

Please let me know if you have any questions.

Re: SnapCreator is not deleting snapshots

tom_taelman — Wed, 15 Jan 2014 12:23:20 GMT

Hi,

Thanks for the info and the work.

Just one small question. Can we migrate / upgrade from the community release to the GA release?

Re: SnapCreator is not deleting snapshots

ktenzer — Wed, 15 Jan 2014 12:46:42 GMT

We dont officially support upgrade from community to GA release. In theory it should work but it isnt something we test and there could be schema changes in community which result in upgrade not working. We will certainly try to help folks here but there is no guarantee it will work and it is not supported today.

In future we hope to improve upgrade and make process more streamlined which would allow us to better support community to GA release. This is something I am really pushing for.

Again regardless of support or not we will try and help you here but we dont recommend upgrading from community release to GA. A better way to do this would be fresh install of GA and copy over configs/logs and re-create users, schedules, etc.

Keith