Data Backup and Recovery
Data Backup and Recovery
Hi,
We have upgraded SnapCreator from 3.6 to 4.1.
In 3.6 everything was working fine. We did not change anything to the config.
Now we have running SnapCreator in 4.1 and see that SnapCreator is not deleting snapshots on the primary. There is also no error in the log files.
We have only a comment line in the logfile:
########## Running Snapshot copy Delete on Primary ##########
########## Running Snapshot copy Delete on Secondary ##########
########## Running Snapshot copy Delete on Secondary NETAPP02 ##########
In the configfile, we have the following config:
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=SERVER15
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_POLICIES=
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_NODELETE=N
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_SNAPSHOT_RETENTION_AGE=daily:2
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y
NTAP_SNAPSHOT_LABEL=
NTAP_SNAPSHOT_CLEANUP=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_SNAPSHOT_RETENTIONS=DAILY:2
NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_METADATA_SNAPSHOT_CREATE_CMD=
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_SNAPSHOT_DISABLE=N
Is there something changed in the upgrade to 4.1?
Why is SnapCreator not deleting my snapshots on the primary? It was working in 3.6 without any problem
I believe this is because the policy is case sensitive and in 4.0 the policies changed to upper case.
I see this in config, I would change NTAP_SNAPSHOT_RETENTIONS=DAILY:2 to NTAP_SNAPSHOT_RETENTIONS=daily:2
NTAP_SNAPSHOT_RETENTIONS=DAILY:2
NTAP_SNAPSHOT_RETENTION_AGE=daily:2
Let us know if this works?
Please pay attention to case sensitivity in regards to policies this can be an issue. I think if you use GUI and create policy they will always be upper case. So if you update something in GUI and are using lower case it can put things out of whack. I recommend switching to upper case, of course if you do then you will have to manually remove the lower case policy snapshots (might be too much work).
Regards,
Keith
Still the same problem.
See complete config file:
CONFIG_TYPE=STANDARD
VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E
META_DATA_VOLUME=
LOG_NUM=30
SNAPDRIVE=Y
SNAPDRIVE_DISCOVERY=Y
LOG_TRACE_ENABLE=Y
USE_PROXY=N
NTAP_PWD_PROTECTION=Y
NTAP_TIMEOUT=60
USE_GLOBAL_CONFIG=N
FEDERATED_APPLICATIONS=
ALLOW_IP_ADDR=N
SC_TMP_DIR=
########################################################################################################################
# CMode Options #
########################################################################################################################
CMODE_CLUSTER_NAME=
CMODE_CLUSTER_USERS=
CMODE_SNAPSHOT_FORCE_DELETE=Y
CMODE_SET=N
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=SERVER15
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_POLICIES=
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_NODELETE=N
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_SNAPSHOT_RETENTION_AGE=DAILY:2
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=Y
NTAP_SNAPSHOT_LABEL=
NTAP_SNAPSHOT_CLEANUP=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_SNAPSHOT_RETENTIONS=DAILY:2
NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_METADATA_SNAPSHOT_CREATE_CMD=
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_SNAPSHOT_DISABLE=N
########################################################################################################################
# SnapVault Options #
########################################################################################################################
NTAP_SNAPVAULT_UPDATE=Y
NTAP_SNAPVAULT_NODELETE=N
SNAPVAULT_QTREE_INCLUDE=
NTAP_SNAPVAULT_RESTORE_WAIT=Y
NTAP_SNAPVAULT_RETENTION_AGE=daily:7
NTAP_SNAPVAULT_SNAPSHOT=N
NTAP_SNAPVAULT_MAX_TRANSFER=
NTAP_SNAPVAULT_WAIT=360
NTAP_SNAPVAULT_RETENTIONS=DAILY:7
SNAPVAULT_VOLUMES=NETAPP01-vif1:SERVER15_D,SERVER15_E
########################################################################################################################
# SnapMirror Options #
########################################################################################################################
NTAP_SNAPMIRROR_WAIT=60
NTAP_SNAPMIRROR_USE_SNAPSHOT=N
NTAP_SNAPMIRROR_MAX_TRANSFER=
SNAPMIRROR_VOLUMES=
SNAPMIRROR_CASCADING_VOLUMES=
NTAP_SNAPMIRROR_CASCADING_UPDATE=N
NTAP_SNAPMIRROR_UPDATE=N
########################################################################################################################
# Clone Options #
########################################################################################################################
NTAP_CLONE_IGROUP_MAP=
NTAP_NFS_EXPORT_ACCESS=
NTAP_LUN_CLONE_RESERVATION=N
NTAP_CLONE_SECONDARY_VOLUMES=
NTAP_NUM_VOL_CLONES=0
NTAP_NFS_EXPORT_PERSISTENT=N
NTAP_CLONE_FOR_BACKUP=Y
NTAP_CIFS_EXPORT_ENABLE=N
NTAP_CLONE_SECONDARY=N
NTAP_NFS_EXPORT_HOST=
NTAP_VOL_CLONE_RESERVE=NONE
########################################################################################################################
# ProtectionManager Options #
########################################################################################################################
NTAP_DFM_DATA_SET=
NTAP_PM_RUN_BACKUP=N
NTAP_PM_UPDATE=N
NTAP_DFM_SNAPSHOT_FORMAT=
########################################################################################################################
# OSSV Options #
########################################################################################################################
NTAP_OSSV_FS_SNAPSHOT=
NTAP_OSSV_FS_SNAPSHOT_CREATE_CMD01=
NTAP_OSSV_HOMEDIR=
NTAP_OSSV_ENABLE=N
########################################################################################################################
# APP Commands #
########################################################################################################################
UMOUNT_CMD01=
ARCHIVE_CMD01=
APP_UNQUIESCE_CMD01=
MOUNT_CMD01=
APP_QUIESCE_CMD01=
########################################################################################################################
# Pre Commands #
########################################################################################################################
PRE_CLONE_CREATE_CMD01=
PRE_RESTORE_CMD01=
PRE_EXIT_CMD01=
PRE_NTAP_CMD01=
PRE_APP_UNQUIESCE_CMD01=
PRE_APP_QUIESCE_CMD01=
PRE_NTAP_CLONE_DELETE_CMD01=
########################################################################################################################
# Post Commands #
########################################################################################################################
POST_NTAP_DATA_TRANSFER_CMD01=
POST_RESTORE_CMD01=
POST_APP_QUIESCE_CMD01=
POST_NTAP_CMD01=
POST_CLONE_CREATE_CMD01=
POST_APP_UNQUIESCE_CMD01=
########################################################################################################################
# Event Options #
########################################################################################################################
SUCCESS_MSG=INFO: Snap Creator finished successfully ( Action: %ACTION )
NTAP_ASUP_ERROR_ENABLE=N
SENDTRAP=
SUCCESS_TRAP=
########################################################################################################################
# Agent Options #
########################################################################################################################
SC_AGENT_UNQUIESCE_TIMEOUT=3605
SC_AGENT_WATCHDOG_ENABLE=N
SC_AGENT=10.10.21.113:9191
SC_AGENT_LOG_ENABLE=Y
SC_AGENT_TIMEOUT=3600
########################################################################################################################
# Clone Target Options #
########################################################################################################################
SC_CLONE_TARGET=
########################################################################################################################
# Application Options #
########################################################################################################################
APP_IGNORE_ERROR=N
APP_NAME=domino
APP_CONF_PERSISTENCE=Y
APP_DEFINED_RESTORE=N
PYTHON_OPTS=
APP_DEFINED_CLONE=N
JAVA_HOME=
PYTHON_HOME=
FS_NAME=
JVM_ARGS=
PERL_HOME=
PERL_OPTS=
PLUGINS_FORWARD=
VALIDATE_VOLUMES=
APP_AUTO_DISCOVERY=N
JAVA_CLASSPATH=
PLUGINS_RESTORE=
APP_DEFINED_BACKUP=N
APP_DEFINED_UMOUNT=N
PLUGINS_REVERSE=
APP_DEFINED_MOUNT=N
########################################################################################################################
# Archive Options #
########################################################################################################################
ARCHIVE_LOG_RECURSIVE_SEARCH=N
ARCHIVE_LOG_DIR=N:\changeinfo\logs
ARCHIVE_LOG_EXT=TXN
ARCHIVE_LOG_RETENTION=2.0
ARCHIVE_LOG_ENABLE=Y
########################################################################################################################
# Plug-In Parameter #
########################################################################################################################
Notes_ExecDirectory=C:\Lotus\Domino
DOMINO_DATABASE_TYPE=0
DOMINO_DATA_PATH=D:\Lotus\Domino\Data
LOTUS=C:\Lotus
DOMINO_RESTORE_DATA_PATH=D:\Lotus\Domino\Data
DOMINO_CHANGE_INFO_PATH=N:\changeinfo
DOMINO_INI_PATH=C:\Lotus\Domino\notes.ini
I have the same for the SnapVault config and there it is working.
Yes if the snapshots on controller have lower case daily and you have upper case DAILY in config and it is case sensitive then your existing snapshots will never be deleted.
Options are continue to use lower case or use upper case and rename snapshots on controller.
I believe this is problem based on info but if you could also provide list of snapshots from controller for this config that would help?
Regards,
Keith
Here is the list of the snapshots.
The latest created snapshot (after changing the config file to uppencase) is also in lowercase
NETAPP01> snap list SERVER15_D
Volume SERVER15_D
working...
%/used %/total date name
---------- ---------- ------------ --------
0% ( 0%) 0% ( 0%) Jan 13 11:10 SERVER15-daily_20140113110938 (snapvault)
5% ( 5%) 1% ( 0%) Jan 12 23:30 SERVER15-daily_20140112233000
10% ( 5%) 1% ( 1%) Jan 11 23:30 SERVER15-daily_20140111233000
14% ( 5%) 2% ( 0%) Jan 10 23:30 SERVER15-daily_20140110233000
NETAPP01> snap list SERVER15_E
Volume SERVER15_E
working...
%/used %/total date name
---------- ---------- ------------ --------
0% ( 0%) 0% ( 0%) Jan 13 11:10 SERVER15-daily_20140113110938 (snapvault)
8% ( 8%) 1% ( 1%) Jan 12 23:30 SERVER15-daily_20140112233000
17% (10%) 3% ( 2%) Jan 11 23:30 SERVER15-daily_20140111233000
24% (10%) 4% ( 2%) Jan 10 23:30 SERVER15-daily_20140110233000
If it was case sensitive SnapCreator should created the latest snapshot with daily in uppercase?
Can you change this to uppercase as well?
NTAP_SNAPVAULT_RETENTION_AGE=daily:7
I just ran a 4.1 config with DAILY as uppercase and the snapshots made had daily lowercase and deletion was working. In addition you can create a new config and then compare what differences are, maybe something else is out of whack?
In addition how did you create this config? It looks based on config format you created a new config in the 4.1 GUI? We have tested upgrade and 3.6 configs should run and work unchanged in 4.0 and 4.1.
Regards,
Keith
I have the same issue with a complete newly created config via the GUI for another server.
This is the config:
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=DCSLHOSA01
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_POLICIES=
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_NODELETE=N
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_SNAPSHOT_RETENTION_AGE=DAILY:5
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N
NTAP_SNAPSHOT_LABEL=
NTAP_SNAPSHOT_CLEANUP=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_SNAPSHOT_RETENTIONS=DAILY:5
NTAP_SNAPSHOT_CREATE_CMD01=sdcli snap create -D D: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02=sdcli snap create -D E: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD03=sdcli snap create -D F: -s %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_METADATA_SNAPSHOT_CREATE_CMD=
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_SNAPSHOT_DISABLE=N
In the log files, I see only this:
########## ARCHIVE COMMANDS ##########
[2014-01-13 14:43:00,570] INFO: Archive commands are not defined
########## Running Snapshot copy Delete on Primary ##########
########## Application cleanup ##########
I look like the function to delete snapshots is not working or not started. All other function have a timestamp and at least INFO but the function Running Snapshot copy Delete on Primary is only logged.
I have passed this on to one of our engineers, he is on EST so should comment on this shortly.
Regards,
Keith
Oh one more things how are you running the backup jobs? Can you give us example? In addition if you can add a debug log of a backup where something isnt deleted that would be great too.
Keith
The backups are started from the scheduler or from the GUI via the drop down menu Actions - Backup and choosing the daily profile
You can find the debug logs of 2 configs with their conf file in attachment
Hello Tom,
I just started reviewing DCSLHOSA01 - and hence few questions around that.
1) Do you have snapvault configured for server DCSLHOSA01 volumes?
2) If so, Can you copy paste the snapvault status output from the NetApp controller for the below volumes?
controller: lhoiststo02
Volumes: DCSLHOSA01_F,DCSLHOSA01_E,DCSLHOSA01_D
3) NTAP_CLONE_FOR_BACKUP=Y
Do you really need this parameter? What is the use case?
The reason I am asking is because, I see this in the debug log.
########## Running storage discovery ##########
[2014-01-13 14:42:01,812] INFO: Skipping Snapshot copy discovery
########## Storage discovery finished successfully ! ##########
There is no snapshot discovery and hence (I believe) is no snapshot deletion on the primary.
4) Could you please change this variable to N (if you don't use that cloning feature) and try a backup again?
NTAP_CLONE_FOR_BACKUP=N
NTAP_CLONE_FOR_BACKUP explained:
There may be use cases where a clone needs to be created right after a snapshot is taken.
Example:
snapshot->lun clone->mount->backup to tape->clone delete
snapshot->vol clone->mount->DR test->clone delete
Hi
The server DCSLHOSA01 has no SnapVault volumes
We did not need the option NTAP_CLONE_FOR_BACKUP. For DCSLHOSA01 the value was already N and for the other server, I have changed it to N
Created a new backup, but still the same issue. No snapshot deleted on primary.
Thanks. Let me test this in the lab and get back to you shortly.
Are you able to list backups for this profile/config?
snapcreator --server <servername> --port 8443 --user <username> --passwd <password> --profile Oracle --config DCSLHOSA01 --action backupList --policy daily --debug --verbose
Please let me know the output.
Thanks.
I will file a defect request on this and have our core developers look into it.
Please give me a day or two and I will get back to you on the progress.
Sorry for the inconvenience.
Dear Tom,
This issue seems to have been already fixed by the 4.1GA release.
I was able to consistently recreate this issue with 4.1 community release.
With 4.1GA release this works perfectly as expected.
Unfortunately you will have to wait for couple of weeks for the 4.1GA code. (as you can read from this thread https://communities.netapp.com/message/123394#123394 )
Please let me know if you have any questions.
Hi,
Thanks for the info and the work.
Just one small question. Can we migrate / upgrade from the community release to the GA release?
We dont officially support upgrade from community to GA release. In theory it should work but it isnt something we test and there could be schema changes in community which result in upgrade not working. We will certainly try to help folks here but there is no guarantee it will work and it is not supported today.
In future we hope to improve upgrade and make process more streamlined which would allow us to better support community to GA release. This is something I am really pushing for.
Again regardless of support or not we will try and help you here but we dont recommend upgrading from community release to GA. A better way to do this would be fresh install of GA and copy over configs/logs and re-create users, schedules, etc.
Keith
I have the exact same issue in Snapcreator 4.1.2 GA release.
Snapshots are created correctly, also a snapcreator Backuplist works perfectly:
C:\Program Files\NetApp\Snap_Creator_Framework\scServer4.1.2>snapcreator --server opsmgr01 --port 8888 --profile Application_servers --config fileserver01_snapdrive_hourly_XDP --action backupList --policy hourly --debug --verbose
########## NetApp Snap Creator Framework 4.1.2 Snapshot copy (Primary) List for vs03_luns:vs03_fileserver01_DM ##########
### Snapshot copy Name ### ### Snapshot copy Timestamp ### ### Busy ###
sc_snapdrive-hourly_20160610080500 Jun 10 2016 08:05:15 false
sc_snapdrive-hourly_20160610120500 Jun 10 2016 12:05:08 false
sc_snapdrive-hourly_20160610160500 Jun 10 2016 16:05:10 false
sc_snapdrive-hourly_20160610190500 Jun 10 2016 19:05:09 false
sc_snapdrive-hourly_20160611080500 Jun 11 2016 08:05:09 false
sc_snapdrive-hourly_20160611120500 Jun 11 2016 12:05:08 false
sc_snapdrive-hourly_20160611160500 Jun 11 2016 16:05:13 false
.....
sc_snapdrive-hourly_20160719123835 Jul 19 2016 12:38:52 false
sc_snapdrive-hourly_20160719125918 Jul 19 2016 12:59:36 false
sc_snapdrive-hourly_20160719131625 Jul 19 2016 13:16:37 false
sc_snapdrive-hourly_20160719132346 Jul 19 2016 13:23:57 false
This is the relevant snapshot part in my config file:
########################################################################################################################
# Snapshot Options #
########################################################################################################################
SNAME=sc_snapdrive
SNAP_TIMESTAMP_ONLY=Y
NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N
NTAP_SNAPSHOT_CREATE_CMD01="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Command Import-Module SnapDrive;New-SdSnapshot -Path "F:,I:" -Snapshot %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD02="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Command Import-Module SnapDrive;New-SdSnapshot -Path G: -Snapshot %SNAME-%SNAP_TYPE_%SNAP_TIME
NTAP_SNAPSHOT_CREATE_CMD03="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Command Import-Module SnapDrive;Set-SdSnapshot -Snapshot %SNAME-%SNAP_TYPE_%SNAP_TIME -StorageSystem vs03_luns -volumename vs03_fileserver01_DM -Snapmirrorlabel hourly
NTAP_SNAPSHOT_CREATE_CMD04="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Command Import-Module SnapDrive;Set-SdSnapshot -Snapshot %SNAME-%SNAP_TYPE_%SNAP_TIME -StorageSystem vs03_luns -volumename vs03_fileserver01_g -Snapmirrorlabel hourly
NTAP_SNAPSHOT_CREATE_CMD05="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Command Import-Module SnapDrive;Write-Debug %SNAME-%SNAP_TYPE_%SNAP_TIME;Invoke-SdSnapMirrorUpdate -SourceStorageSystem vs03_luns -SourceVolumename vs03_fileserver01_DM -DestinationStorageSystem vs31_backup -DestinationStorageVolume vs03_fileserver01_DM_vault
NTAP_SNAPSHOT_CREATE_CMD06="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Command Import-Module SnapDrive;Write-Debug %SNAME-%SNAP_TYPE_%SNAP_TIME;Invoke-SdSnapMirrorUpdate -SourceStorageSystem vs03_luns -SourceVolumename vs03_fileserver01_g -DestinationStorageSystem vs31_backup -DestinationStorageVolume vs03_fileserver01_g_vault
NTAP_CONSISTENCY_GROUP_SNAPSHOT=N
NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N
NTAP_USE_EXTERNAL_SNAPSHOT=N
NTAP_SNAPSHOT_POLICIES=
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N
NTAP_SNAPSHOT_DISABLE=Y
NTAP_METADATA_SNAPSHOT_CREATE_CMD01=
NTAP_SNAPSHOT_RETENTION_AGE=
NTAP_SNAPSHOT_CLEANUP=Y
NTAP_SNAPSHOT_NODELETE=N
NTAP_SNAPSHOT_LABEL=
ALLOW_DUPLICATE_SNAME=Y
NTAP_SNAPSHOT_RETENTIONS=HOURLY:12
NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N
NTAP_EXTERNAL_SNAPSHOT_REGEX=.*
NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM
In my debug log I see the following when running the Backup with policy hourly:
########## Running Snapshot copy Delete on Primary ##########
[2016-07-19 13:17:16,845] INFO: Application not defined. Skipping cleanup task
So nothing gets deleted and my backup start failing as soon as 255 snapshots have been reached on the volume.
PS: I also tried setting NTAP_SNAPSHOT_RETENTIONS=HOURLY:12 to NTAP_SNAPSHOT_RETENTIONS=hourly:12 without result.
Any idea ?
Grtz,
Tom De Wit
I found the solution to my problem.
The parameter NTAP_SNAPSHOT_DISABLE has to be 'N' for Snapcreator to clean up snapshots. If this parameter is enabled, Snapcreator will just skip the snapshot delete process
I was in the impression that this parameter needed to be 'Y' because I was taking snapshots externally through Snapdrive commands, so I didn't want a normal ZAPI snapshot create also. This was a difficult one to find because snapshot creation was working and only the snapshot deletion was being skipped (not honouring my policy).
Grtz,
Tom