Data Backup and Recovery Discussions
Data Backup and Recovery Discussions
Hi,
We have upgraded SnapCreator from 3.6 to 4.1.
In 3.6 everything was working fine. We did not change anything to the config.
Now we have running SnapCreator in 4.1 and see that SnapCreator is not deleting snapshots on the primary. There is also no error in the log files.
We have only a comment line in the logfile:
########## Running Snapshot copy Delete on Primary ##########
########## Running Snapshot copy Delete on Secondary ##########
########## Running Snapshot copy Delete on Secondary NETAPP02 ##########
In the configfile, we have the following config:
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=SERVER15
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_POLICIES=
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_NODELETE=N
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_SNAPSHOT_RETENTION_AGE=daily:2
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y
NTAP_SNAPSHOT_LABEL=
NTAP_SNAPSHOT_CLEANUP=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_SNAPSHOT_RETENTIONS=DAILY:2
NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_METADATA_SNAPSHOT_CREATE_CMD=
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_SNAPSHOT_DISABLE=N
Is there something changed in the upgrade to 4.1?
Why is SnapCreator not deleting my snapshots on the primary? It was working in 3.6 without any problem
I believe this is because the policy is case sensitive and in 4.0 the policies changed to upper case.
I see this in config, I would change NTAP_SNAPSHOT_RETENTIONS=DAILY:2 to NTAP_SNAPSHOT_RETENTIONS=daily:2
NTAP_SNAPSHOT_RETENTIONS=DAILY:2
NTAP_SNAPSHOT_RETENTION_AGE=daily:2
Let us know if this works?
Please pay attention to case sensitivity in regards to policies this can be an issue. I think if you use GUI and create policy they will always be upper case. So if you update something in GUI and are using lower case it can put things out of whack. I recommend switching to upper case, of course if you do then you will have to manually remove the lower case policy snapshots (might be too much work).
Regards,
Keith
Still the same problem.
See complete config file:
CONFIG_TYPE=STANDARD
VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E
META_DATA_VOLUME=
LOG_NUM=30
SNAPDRIVE=Y
SNAPDRIVE_DISCOVERY=Y
LOG_TRACE_ENABLE=Y
USE_PROXY=N
NTAP_PWD_PROTECTION=Y
NTAP_TIMEOUT=60
USE_GLOBAL_CONFIG=N
FEDERATED_APPLICATIONS=
ALLOW_IP_ADDR=N
SC_TMP_DIR=
########################################################################################################################
# CMode Options #
########################################################################################################################
CMODE_CLUSTER_NAME=
CMODE_CLUSTER_USERS=
CMODE_SNAPSHOT_FORCE_DELETE=Y
CMODE_SET=N
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=SERVER15
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_POLICIES=
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_NODELETE=N
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_SNAPSHOT_RETENTION_AGE=DAILY:2
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y
NTAP_SNAPSHOT_LABEL=
NTAP_SNAPSHOT_CLEANUP=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_SNAPSHOT_RETENTIONS=DAILY:2
NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_METADATA_SNAPSHOT_CREATE_CMD=
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_SNAPSHOT_DISABLE=N
########################################################################################################################
# SnapVault Options #
########################################################################################################################
NTAP_SNAPVAULT_UPDATE=Y
NTAP_SNAPVAULT_NODELETE=N
SNAPVAULT_QTREE_INCLUDE=
NTAP_SNAPVAULT_RESTORE_WAIT=Y
NTAP_SNAPVAULT_RETENTION_AGE=daily:7
NTAP_SNAPVAULT_SNAPSHOT=N
NTAP_SNAPVAULT_MAX_TRANSFER=
NTAP_SNAPVAULT_WAIT=360
NTAP_SNAPVAULT_RETENTIONS=DAILY:7
SNAPVAULT_VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E
########################################################################################################################
# SnapMirror Options #
########################################################################################################################
NTAP_SNAPMIRROR_WAIT=60
NTAP_SNAPMIRROR_USE_SNAPSHOT=N
NTAP_SNAPMIRROR_MAX_TRANSFER=
SNAPMIRROR_VOLUMES=
SNAPMIRROR_CASCADING_VOLUMES=
NTAP_SNAPMIRROR_CASCADING_UPDATE=N
NTAP_SNAPMIRROR_UPDATE=N
########################################################################################################################
# Clone Options #
########################################################################################################################
NTAP_CLONE_IGROUP_MAP=
NTAP_NFS_EXPORT_ACCESS=
NTAP_LUN_CLONE_RESERVATION=N
NTAP_CLONE_SECONDARY_VOLUMES=
NTAP_NUM_VOL_CLONES=0
NTAP_NFS_EXPORT_PERSISTENT=N
NTAP_CLONE_FOR_BACKUP=Y
NTAP_CIFS_EXPORT_ENABLE=N
NTAP_CLONE_SECONDARY=N
NTAP_NFS_EXPORT_HOST=
NTAP_VOL_CLONE_RESERVE=NONE
########################################################################################################################
# ProtectionManager Options #
########################################################################################################################
NTAP_DFM_DATA_SET=
NTAP_PM_RUN_BACKUP=N
NTAP_PM_UPDATE=N
NTAP_DFM_SNAPSHOT_FORMAT=
########################################################################################################################
# OSSV Options #
########################################################################################################################
NTAP_OSSV_FS_SNAPSHOT=
NTAP_OSSV_FS_SNAPSHOT_CREATE_CMD01=
NTAP_OSSV_HOMEDIR=
NTAP_OSSV_ENABLE=N
########################################################################################################################
# APP Commands #
########################################################################################################################
UMOUNT_CMD01=
ARCHIVE_CMD01=
APP_UNQUIESCE_CMD01=
MOUNT_CMD01=
APP_QUIESCE_CMD01=
########################################################################################################################
# Pre Commands #
########################################################################################################################
PRE_CLONE_CREATE_CMD01=
PRE_RESTORE_CMD01=
PRE_EXIT_CMD01=
PRE_NTAP_CMD01=
PRE_APP_UNQUIESCE_CMD01=
PRE_APP_QUIESCE_CMD01=
PRE_NTAP_CLONE_DELETE_CMD01=
########################################################################################################################
# Post Commands #
########################################################################################################################
POST_NTAP_DATA_TRANSFER_CMD01=
POST_RESTORE_CMD01=
POST_APP_QUIESCE_CMD01=
POST_NTAP_CMD01=
POST_CLONE_CREATE_CMD01=
POST_APP_UNQUIESCE_CMD01=
########################################################################################################################
# Event Options #
########################################################################################################################
SUCCESS_MSG=INFO: Snap Creator finished successfully ( Action: %ACTION )
NTAP_ASUP_ERROR_ENABLE=N
SENDTRAP=
SUCCESS_TRAP=
########################################################################################################################
# Agent Options #
########################################################################################################################
SC_AGENT_UNQUIESCE_TIMEOUT=3605
SC_AGENT_WATCHDOG_ENABLE=N
SC_AGENT=10.10.21.113:9191
SC_AGENT_LOG_ENABLE=Y
SC_AGENT_TIMEOUT=3600
########################################################################################################################
# Clone Target Options #
########################################################################################################################
SC_CLONE_TARGET=
########################################################################################################################
# Application Options #
########################################################################################################################
APP_IGNORE_ERROR=N
APP_NAME=domino
APP_CONF_PERSISTENCE=Y
APP_DEFINED_RESTORE=N
PYTHON_OPTS=
APP_DEFINED_CLONE=N
JAVA_HOME=
PYTHON_HOME=
FS_NAME=
JVM_ARGS=
PERL_HOME=
PERL_OPTS=
PLUGINS_FORWARD=
VALIDATE_VOLUMES=
APP_AUTO_DISCOVERY=N
JAVA_CLASSPATH=
PLUGINS_RESTORE=
APP_DEFINED_BACKUP=N
APP_DEFINED_UMOUNT=N
PLUGINS_REVERSE=
APP_DEFINED_MOUNT=N
########################################################################################################################
# Archive Options #
########################################################################################################################
ARCHIVE_LOG_RECURSIVE_SEARCH=N
ARCHIVE_LOG_DIR=N:\changeinfo\logs
ARCHIVE_LOG_EXT=TXN
ARCHIVE_LOG_RETENTION=2.0
ARCHIVE_LOG_ENABLE=Y
########################################################################################################################
# Plug-In Parameter #
########################################################################################################################
Notes_ExecDirectory=C:\Lotus\Domino
DOMINO_DATABASE_TYPE=0
DOMINO_DATA_PATH=D:\Lotus\Domino\Data
LOTUS=C:\Lotus
DOMINO_RESTORE_DATA_PATH=D:\Lotus\Domino\Data
DOMINO_CHANGE_INFO_PATH=N:\changeinfo
DOMINO_INI_PATH=C:\Lotus\Domino\notes.ini
I have the same for the SnapVault config and there it is working.
Yes if the snapshots on controller have lower case daily and you have upper case DAILY in config and it is case sensitive then your existing snapshots will never be deleted.
Options are continue to use lower case or use upper case and rename snapshots on controller.
I believe this is problem based on info but if you could also provide list of snapshots from controller for this config that would help?
Regards,
Keith
Here is the list of the snapshots.
The latest created snapshot (after changing the config file to uppencase) is also in lowercase
NETAPP01> snap list SERVER15_D
Volume SERVER15_D
working...
%/used %/total date name
---------- ---------- ------------ --------
0% ( 0%) 0% ( 0%) Jan 13 11:10 SERVER15-daily_20140113110938 (snapvault)
5% ( 5%) 1% ( 0%) Jan 12 23:30 SERVER15-daily_20140112233000
10% ( 5%) 1% ( 1%) Jan 11 23:30 SERVER15-daily_20140111233000
14% ( 5%) 2% ( 0%) Jan 10 23:30 SERVER15-daily_20140110233000
NETAPP01> snap list SERVER15_E
Volume SERVER15_E
working...
%/used %/total date name
---------- ---------- ------------ --------
0% ( 0%) 0% ( 0%) Jan 13 11:10 SERVER15-daily_20140113110938 (snapvault)
8% ( 8%) 1% ( 1%) Jan 12 23:30 SERVER15-daily_20140112233000
17% (10%) 3% ( 2%) Jan 11 23:30 SERVER15-daily_20140111233000
24% (10%) 4% ( 2%) Jan 10 23:30 SERVER15-daily_20140110233000
If it was case sensitive SnapCreator should created the latest snapshot with daily in uppercase?
Can you change this to uppercase as well?
NTAP_SNAPVAULT_RETENTION_AGE=daily:7
I just ran a 4.1 config with DAILY as uppercase and the snapshots made had daily lowercase and deletion was working. In addition you can create a new config and then compare what differences are, maybe something else is out of whack?
In addition how did you create this config? It looks based on config format you created a new config in the 4.1 GUI? We have tested upgrade and 3.6 configs should run and work unchanged in 4.0 and 4.1.
Regards,
Keith
I have the same issue with a complete newly created config via the GUI for another server.
This is the config:
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=DCSLHOSA01
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_POLICIES=
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_NODELETE=N
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_SNAPSHOT_RETENTION_AGE=DAILY:5
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N
NTAP_SNAPSHOT_LABEL=
NTAP_SNAPSHOT_CLEANUP=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_SNAPSHOT_RETENTIONS=DAILY:5
NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD03=sdcli snap create -D F: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_METADATA_SNAPSHOT_CREATE_CMD=
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_SNAPSHOT_DISABLE=N
In the log files, I see only this:
########## ARCHIVE COMMANDS ##########
[2014-01-13 14:43:00,570] INFO: Archive commands are not defined
########## Running Snapshot copy Delete on Primary ##########
########## Application cleanup ##########
I look like the function to delete snapshots is not working or not started. All other function have a timestamp and at least INFO but the function Running Snapshot copy Delete on Primary is only logged.
I have passed this on to one of our engineers, he is on EST so should comment on this shortly.
Regards,
Keith
Oh one more things how are you running the backup jobs? Can you give us example? In addition if you can add a debug log of a backup where something isnt deleted that would be great too.
Keith
The backups are started from the scheduler or from the GUI via the drop down menu Actions - Backup and choosing the daily profile
You can find the debug logs of 2 configs with their conf file in attachment
Hello Tom,
I just started reviewing DCSLHOSA01 - and hence few questions around that.
1) Do you have snapvault configured for server DCSLHOSA01 volumes?
2) If so, Can you copy paste the snapvault status output from the NetApp controller for the below volumes?
controller: lhoiststo02
Volumes: DCSLHOSA01_F,DCSLHOSA01_E,DCSLHOSA01_D
3) NTAP_CLONE_FOR_BACKUP=Y
Do you really need this parameter? What is the use case?
The reason I am asking is because, I see this in the debug log.
########## Running storage discovery ##########
[2014-01-13 14:42:01,812] INFO: Skipping Snapshot copy discovery
########## Storage discovery finished successfully ! ##########
There is no snapshot discovery and hence (I believe) is no snapshot deletion on the primary.
4) Could you please change this variable to N (if you don't use that cloning feature) and try a backup again?
NTAP_CLONE_FOR_BACKUP=N
NTAP_CLONE_FOR_BACKUP explained:
There may be use cases where a clone needs to be created right after a snapshot is taken.
Example:
snapshot->lun clone->mount->backup to tape->clone delete
snapshot->vol clone->mount->DR test->clone delete
Hi
The server DCSLHOSA01 has no SnapVault volumes
We did not need the option NTAP_CLONE_FOR_BACKUP. For DCSLHOSA01 the value was already N and for the other server, I have changed it to N
Created a new backup, but still the same issue. No snapshot deleted on primary.
Thanks. Let me test this in the lab and get back to you shortly.
Are you able to list backups for this profile/config?
snapcreator --server <servername> --port 8443 --user <username> --passwd <password> --profile Oracle --config DCSLHOSA01 --action backupList --policy daily --debug --verbose
Please let me know the output.
Thanks.
I am able to reproduce this behavior in the lab.
I am looking into this now.
I will keep you posted soon.