SnapCreator is not deleting snapshots

tom_taelman · ‎2014-01-13

Hi,

We have upgraded SnapCreator from 3.6 to 4.1.

In 3.6 everything was working fine. We did not change anything to the config.

Now we have running SnapCreator in 4.1 and see that SnapCreator is not deleting snapshots on the primary. There is also no error in the log files.

We have only a comment line in the logfile:

########## Running Snapshot copy Delete on Primary ##########

########## Running Snapshot copy Delete on Secondary ##########

########## Running Snapshot copy Delete on Secondary NETAPP02 ##########

In the configfile, we have the following config:

########################################################################################################################

# Snapshot Options #

########################################################################################################################

SNAME=SERVER15

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RETENTION_AGE=daily:2

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y

NTAP_SNAPSHOT_LABEL=

NTAP_SNAPSHOT_CLEANUP=N

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_RETENTIONS=DAILY:2

NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_DISABLE=N

Is there something changed in the upgrade to 4.1?

Why is SnapCreator not deleting my snapshots on the primary? It was working in 3.6 without any problem

ktenzer · ‎2014-01-13

I believe this is because the policy is case sensitive and in 4.0 the policies changed to upper case.

I see this in config, I would change NTAP_SNAPSHOT_RETENTIONS=DAILY:2 to NTAP_SNAPSHOT_RETENTIONS=daily:2

NTAP_SNAPSHOT_RETENTIONS=DAILY:2

NTAP_SNAPSHOT_RETENTION_AGE=daily:2

Let us know if this works?

Please pay attention to case sensitivity in regards to policies this can be an issue. I think if you use GUI and create policy they will always be upper case. So if you update something in GUI and are using lower case it can put things out of whack. I recommend switching to upper case, of course if you do then you will have to manually remove the lower case policy snapshots (might be too much work).

Regards,

Keith

tom_taelman · ‎2014-01-13

Still the same problem.

See complete config file:

CONFIG_TYPE=STANDARD

VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E

META_DATA_VOLUME=

LOG_NUM=30

SNAPDRIVE=Y

SNAPDRIVE_DISCOVERY=Y

LOG_TRACE_ENABLE=Y

USE_PROXY=N

NTAP_PWD_PROTECTION=Y

NTAP_TIMEOUT=60

USE_GLOBAL_CONFIG=N

FEDERATED_APPLICATIONS=

ALLOW_IP_ADDR=N

SC_TMP_DIR=

########################################################################################################################

# CMode Options #

########################################################################################################################

CMODE_CLUSTER_NAME=

CMODE_CLUSTER_USERS=

CMODE_SNAPSHOT_FORCE_DELETE=Y

CMODE_SET=N

########################################################################################################################

# Snapshot Options #

########################################################################################################################

SNAME=SERVER15

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RETENTION_AGE=DAILY:2

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y

NTAP_SNAPSHOT_LABEL=

NTAP_SNAPSHOT_CLEANUP=N

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_RETENTIONS=DAILY:2

NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_DISABLE=N

########################################################################################################################

# SnapVault Options #

########################################################################################################################

NTAP_SNAPVAULT_UPDATE=Y

NTAP_SNAPVAULT_NODELETE=N

SNAPVAULT_QTREE_INCLUDE=

NTAP_SNAPVAULT_RESTORE_WAIT=Y

NTAP_SNAPVAULT_RETENTION_AGE=daily:7

NTAP_SNAPVAULT_SNAPSHOT=N

NTAP_SNAPVAULT_MAX_TRANSFER=

NTAP_SNAPVAULT_WAIT=360

NTAP_SNAPVAULT_RETENTIONS=DAILY:7

SNAPVAULT_VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E

########################################################################################################################

# SnapMirror Options #

########################################################################################################################

NTAP_SNAPMIRROR_WAIT=60

NTAP_SNAPMIRROR_USE_SNAPSHOT=N

NTAP_SNAPMIRROR_MAX_TRANSFER=

SNAPMIRROR_VOLUMES=

SNAPMIRROR_CASCADING_VOLUMES=

NTAP_SNAPMIRROR_CASCADING_UPDATE=N

NTAP_SNAPMIRROR_UPDATE=N

########################################################################################################################

# Clone Options #

########################################################################################################################

NTAP_CLONE_IGROUP_MAP=

NTAP_NFS_EXPORT_ACCESS=

NTAP_LUN_CLONE_RESERVATION=N

NTAP_CLONE_SECONDARY_VOLUMES=

NTAP_NUM_VOL_CLONES=0

NTAP_NFS_EXPORT_PERSISTENT=N

NTAP_CLONE_FOR_BACKUP=Y

NTAP_CIFS_EXPORT_ENABLE=N

NTAP_CLONE_SECONDARY=N

NTAP_NFS_EXPORT_HOST=

NTAP_VOL_CLONE_RESERVE=NONE

########################################################################################################################

# ProtectionManager Options #

########################################################################################################################

NTAP_DFM_DATA_SET=

NTAP_PM_RUN_BACKUP=N

NTAP_PM_UPDATE=N

NTAP_DFM_SNAPSHOT_FORMAT=

########################################################################################################################

# OSSV Options #

########################################################################################################################

NTAP_OSSV_FS_SNAPSHOT=

NTAP_OSSV_FS_SNAPSHOT_CREATE_CMD01=

NTAP_OSSV_HOMEDIR=

NTAP_OSSV_ENABLE=N

########################################################################################################################

# APP Commands #

########################################################################################################################

UMOUNT_CMD01=

ARCHIVE_CMD01=

APP_UNQUIESCE_CMD01=

MOUNT_CMD01=

APP_QUIESCE_CMD01=

########################################################################################################################

# Pre Commands #

########################################################################################################################

PRE_CLONE_CREATE_CMD01=

PRE_RESTORE_CMD01=

PRE_EXIT_CMD01=

PRE_NTAP_CMD01=

PRE_APP_UNQUIESCE_CMD01=

PRE_APP_QUIESCE_CMD01=

PRE_NTAP_CLONE_DELETE_CMD01=

########################################################################################################################

# Post Commands #

########################################################################################################################

POST_NTAP_DATA_TRANSFER_CMD01=

POST_RESTORE_CMD01=

POST_APP_QUIESCE_CMD01=

POST_NTAP_CMD01=

POST_CLONE_CREATE_CMD01=

POST_APP_UNQUIESCE_CMD01=

########################################################################################################################

# Event Options #

########################################################################################################################

SUCCESS_MSG=INFO: Snap Creator finished successfully ( Action: %ACTION )

NTAP_ASUP_ERROR_ENABLE=N

SENDTRAP=

SUCCESS_TRAP=

########################################################################################################################

# Agent Options #

########################################################################################################################

SC_AGENT_UNQUIESCE_TIMEOUT=3605

SC_AGENT_WATCHDOG_ENABLE=N

SC_AGENT=10.10.21.113:9191

SC_AGENT_LOG_ENABLE=Y

SC_AGENT_TIMEOUT=3600

########################################################################################################################

# Clone Target Options #

########################################################################################################################

SC_CLONE_TARGET=

########################################################################################################################

# Application Options #

########################################################################################################################

APP_IGNORE_ERROR=N

APP_NAME=domino

APP_CONF_PERSISTENCE=Y

APP_DEFINED_RESTORE=N

PYTHON_OPTS=

APP_DEFINED_CLONE=N

JAVA_HOME=

PYTHON_HOME=

FS_NAME=

JVM_ARGS=

PERL_HOME=

PERL_OPTS=

PLUGINS_FORWARD=

VALIDATE_VOLUMES=

APP_AUTO_DISCOVERY=N

JAVA_CLASSPATH=

PLUGINS_RESTORE=

APP_DEFINED_BACKUP=N

APP_DEFINED_UMOUNT=N

PLUGINS_REVERSE=

APP_DEFINED_MOUNT=N

########################################################################################################################

# Archive Options #

########################################################################################################################

ARCHIVE_LOG_RECURSIVE_SEARCH=N

ARCHIVE_LOG_DIR=N:\changeinfo\logs

ARCHIVE_LOG_EXT=TXN

ARCHIVE_LOG_RETENTION=2.0

ARCHIVE_LOG_ENABLE=Y

########################################################################################################################

# Plug-In Parameter #

########################################################################################################################

Notes_ExecDirectory=C:\Lotus\Domino

DOMINO_DATABASE_TYPE=0

DOMINO_DATA_PATH=D:\Lotus\Domino\Data

LOTUS=C:\Lotus

DOMINO_RESTORE_DATA_PATH=D:\Lotus\Domino\Data

DOMINO_CHANGE_INFO_PATH=N:\changeinfo

DOMINO_INI_PATH=C:\Lotus\Domino\notes.ini

I have the same for the SnapVault config and there it is working.

ktenzer · ‎2014-01-13

Yes if the snapshots on controller have lower case daily and you have upper case DAILY in config and it is case sensitive then your existing snapshots will never be deleted.

Options are continue to use lower case or use upper case and rename snapshots on controller.

I believe this is problem based on info but if you could also provide list of snapshots from controller for this config that would help?

Regards,

Keith

tom_taelman · ‎2014-01-13

Here is the list of the snapshots.

The latest created snapshot (after changing the config file to uppencase) is also in lowercase

NETAPP01> snap list SERVER15_D
Volume SERVER15_D
working...
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Jan 13 11:10  SERVER15-daily_20140113110938 (snapvault)
  5% ( 5%)    1% ( 0%)  Jan 12 23:30  SERVER15-daily_20140112233000
 10% ( 5%)    1% ( 1%)  Jan 11 23:30  SERVER15-daily_20140111233000
 14% ( 5%)    2% ( 0%)  Jan 10 23:30  SERVER15-daily_20140110233000
NETAPP01> snap list SERVER15_E
Volume SERVER15_E
working...
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Jan 13 11:10  SERVER15-daily_20140113110938 (snapvault)
  8% ( 8%)    1% ( 1%)  Jan 12 23:30  SERVER15-daily_20140112233000
 17% (10%)    3% ( 2%)  Jan 11 23:30  SERVER15-daily_20140111233000
 24% (10%)    4% ( 2%)  Jan 10 23:30  SERVER15-daily_20140110233000

If it was case sensitive SnapCreator should created the latest snapshot with daily in uppercase?

ktenzer · ‎2014-01-13

Can you change this to uppercase as well?

NTAP_SNAPVAULT_RETENTION_AGE=daily:7

I just ran a 4.1 config with DAILY as uppercase and the snapshots made had daily lowercase and deletion was working. In addition you can create a new config and then compare what differences are, maybe something else is out of whack?

In addition how did you create this config? It looks based on config format you created a new config in the 4.1 GUI? We have tested upgrade and 3.6 configs should run and work unchanged in 4.0 and 4.1.

Regards,

Keith

tom_taelman · ‎2014-01-13

I have the same issue with a complete newly created config via the GUI for another server.

This is the config:

########################################################################################################################

# Snapshot Options #

########################################################################################################################

SNAME=DCSLHOSA01

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RETENTION_AGE=DAILY:5

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

NTAP_SNAPSHOT_LABEL=

NTAP_SNAPSHOT_CLEANUP=N

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_RETENTIONS=DAILY:5

NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_SNAPSHOT_CREATE_CMD03=sdcli snap create -D F: -s %SNAME-%SNAP_TYPE_%SNAP_TIME

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_DISABLE=N

In the log files, I see only this:

########## ARCHIVE COMMANDS ##########

[2014-01-13 14:43:00,570] INFO: Archive commands are not defined

########## Running Snapshot copy Delete on Primary ##########

########## Application cleanup ##########

I look like the function to delete snapshots is not working or not started. All other function have a timestamp and at least INFO but the function Running Snapshot copy Delete on Primary is only logged.

ktenzer · ‎2014-01-13

I have passed this on to one of our engineers, he is on EST so should comment on this shortly.

Regards,

Keith

ktenzer · ‎2014-01-13

Oh one more things how are you running the backup jobs? Can you give us example? In addition if you can add a debug log of a backup where something isnt deleted that would be great too.

Keith

tom_taelman · ‎2014-01-13

The backups are started from the scheduler or from the GUI via the drop down menu Actions - Backup and choosing the daily profile

You can find the debug logs of 2 configs with their conf file in attachment

sivar · ‎2014-01-13

Hello Tom,

I just started reviewing DCSLHOSA01 - and hence few questions around that.

1) Do you have snapvault configured for server DCSLHOSA01 volumes?

2) If so, Can you copy paste the snapvault status output from the NetApp controller for the below volumes?

controller: lhoiststo02

Volumes: DCSLHOSA01_F,DCSLHOSA01_E,DCSLHOSA01_D

3) NTAP_CLONE_FOR_BACKUP=Y

Do you really need this parameter? What is the use case?

The reason I am asking is because, I see this in the debug log.

########## Running storage discovery ##########

[2014-01-13 14:42:01,812] INFO: Skipping Snapshot copy discovery

########## Storage discovery finished successfully ! ##########

There is no snapshot discovery and hence (I believe) is no snapshot deletion on the primary.

4) Could you please change this variable to N (if you don't use that cloning feature) and try a backup again?

NTAP_CLONE_FOR_BACKUP=N

NTAP_CLONE_FOR_BACKUP explained:

There may be use cases where a clone needs to be created right after a snapshot is taken.

Example:

snapshot->lun clone->mount->backup to tape->clone delete

snapshot->vol clone->mount->DR test->clone delete

tom_taelman · ‎2014-01-13

Hi

The server DCSLHOSA01 has no SnapVault volumes

We did not need the option NTAP_CLONE_FOR_BACKUP. For DCSLHOSA01 the value was already N and for the other server, I have changed it to N

Created a new backup, but still the same issue. No snapshot deleted on primary.

sivar · ‎2014-01-13

Thanks. Let me test this in the lab and get back to you shortly.

sivar · ‎2014-01-13

Are you able to list backups for this profile/config?

snapcreator --server <servername> --port 8443 --user <username> --passwd <password> --profile Oracle --config DCSLHOSA01 --action backupList --policy daily --debug --verbose

Please let me know the output.

Thanks.

sivar · ‎2014-01-13

I am able to reproduce this behavior in the lab.

I am looking into this now.

I will keep you posted soon.

tom_taelman · ‎2014-01-14

I was able to produce the list with the backups for this config without any problem.

See attachment.

Do you have more news for me as you was able to reproduce our problem into the lab?

SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots

Re: SnapCreator is not deleting snapshots