IBM Domino Plugin for Snap Creator and the _recent option

Recently there have been several people asking about the reasons why the Domino plugin does not allow the use of the _recent timestamp option.

I thought I'd put together a quick blog post on this topic.

 

Even if you don't care specifically about the Domino plugin this may be worth a read as the "What can I do Today" section of this post gives some good information on how to use the Snap Creator PRE/POST commands to workaround this issue in a way that anyone can  use, regardless of the plugin in use.

 

_recent is a simple way to quickly determine what the newest Snapshot copy in your environment is.  That also makes it easy to set a daily job to send the _recent Snapshot copy to tape.

 

Unfortunately, due to the way that the Domino APIs work, the _recent option is disabled when using the Domino plugin.

We have had a request for well over a year to enable the _recent option in the Domino plugin and it is something we are considering for a future enhancement, but it will require significant coding - potentially a major rewrite of the plugin.  It is something in consideration, but the work has not been completely scoped or assigned to a release. 

 

Adding support for _recent in the Domino plugin was recently posted in the idea section and we will be watching the feedback - to date we assume that only a very small number of customers are affected, and those that are can take advantage of a workaround that will be discussed at the end of this post.

 

Why is _recent disabled with the Domino plugin?

 

First, keep in mind that Domino is a series of small databases - it is typical to have hundreds or thousands of Domino databases in an environment.

For example, If you have 1000 users you will have 1000 databases - one per user (not to mention all of the other databases in the environment required to keep Domino working).

 

Also keep in mind that the Domino plugin requires a changeinfo directory in which to store information related to the backup.  This directory is specified in the Snap Creator configuration file.

 

The Domino plugin utilizes the IBM Domino APIs for quiesce and unquiesce of the Domino environment.

During the quiesce process a marker is placed into the database signifying where the backup starts

During the unquiesce period a .info file is written for each database that logs the information written to the database betweeen quiesce and unquiesce.

These .info files are stored in the changeinfo directory, organized in folders with the same name as the Snapshot copy.

 

For example, if I have a snapshot named: lyon-daily_20131113123000

I will also have within my changeinfo directory a subdirectory created with the same name: lyon-daily_20131113123000

Within that subdirectory the .info files are stored, one per database.

 

Since the snapshot name and the changeinfo directory are named the same, when a restore process is started the plugin knows that the selected snapshot is lyon-daily_20131113123000.  It also knows that the .info files needed for restore are located in the changeinfo file in a subdirectory with the same name as the Snapshot copy.

 

Restore is fairly simple in this case and there is no margin for error.

 

If we allow the _recent option, the .info files will be stored in the _recent folder - this will be required to restore from the most recent snapshot.

ONTAP knows to rename the snapshot to the timestamp when using the _recent option.  This is built into ONTAP.

Snap Creator would then need to rename the _recent directory in changeinfo to the same name as the renamed snapshot.  This would need to be done after the snapshot copy is renamed, but before the unquiesce step where the files are written.

The risk is that if an error occurs in Snap Creator before the changeinfo subdirectory is renamed, but after the Snapshot copy is renamed then the data is essentially in a non recoverable state.

There is also a chance that if the Snapshot copy is renamed, but the changeinfo directory is not, the next time we attempt to write files to _recent the files may already exist and as a result new files may not be written OR new files will be written and the old files be lost. 

Either way this potentially introduces a risk to our ability to recover the Domino data.  As such we have not enabled the _recent option in an attempt to preserve the recoverabiltiy of the Snapshot copy.

 

I need _recent - What can I do today?

 

Fortunately, Snap Creator is a Framework and allows us lots of flexibility.

The need is to have a snapshot copy with a very specific name so that you can send it to tape.

This Snapshot will not be used for Snap Creator recovery and only needs to live for a very short period of time since once you send it to tape it is no longer needed.

 

Rather then trying to mess around with rename scripts to change the name of the Snap Creator Snapshot copy (and potentially cause problems with your restore ability), just take a second Snapshot copy and name it whatever you want!

 

The below is from an email I sent to a partner earlier this year to meet this exact need. 

This details the steps to create a PRE_APP_UNQUIESCE command to take a second Snapshot copy while Domino is still in a quiesced state.

In the example SnapDrive for Windows is used since that is what the customer was using, but any command can be called.

 

Note that the PRE/POST commands are available in the config file, but you will need to edit the config file with a text editor - these parameters are not available in the GUI.

In Snap Creator 3.6 the config files are located in:  /install/path/scServer3.6/configs

[On Windows the default path is: C:\Program Files\NetApp\NetApp_Snap_Creator_Framework\scServer3.6\configs]

In Snap Creator 4.0 the config files are located in: /install/path/scServer4.0/engine/configs

[On Windows the default path is: C:\Program Files\NetApp\Snap_Creator_Framework\scServer4.0.0\engine\configs]

 

Within the configs directory you will see a subfolder for each profile within your Snap Creator environment.

The config files are located with the profile subdirectory and end with .conf.

Open the config file with your text editor of choice and make the changes within the config.

Once you save your settings, if you use the GUI and you have the config file currently open you will need to click the Refresh button on the GUI to get the latest version of the config file with the changes.

 

Also keep in mind that any commands you add into the config file will need to be allowed to be executed on the agent.

This is controlled by a file located in the Snap Creator Agent called agent.conf. [/install/path/scAgent/config/agent.conf]

Simply place the command you are running into the agent.conf file.

For example, to run the sdcli command you would add the following line to the agent.conf file:

command: sdcli.exe

 

The command does not need to be fully qualified.  One command per line, and notice that there is a space after the ":".

Add as many commands as you need.

 

Hopefully this is helpful to you and meets your needs - feel free to post comments below if you have any questions about these steps!

 

--------------------------------------------------------------------------------------------------------------------------------------------------------

You should be able to just call an additional command to take a snapshot and format the name however you like.

Assuming that you are using SnapDrive for Windows you can just call the appropriate sdcli.exe commands.

 

For example: "C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s %SNAME-%SNAP_TYPE_recent -D F

 

Assuming that you are using a Snap Creator config with the name domino01 using the daily policy this would take a snapshot named domino01-daily_recent

This would make sure the snapshot name is consistent for the purposes of sending the snapshot copy to tape.

 

Likely you would want to do something like this:

 

PRE_APP_QUIESCE_CMD01="C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap delete -d F -s %SNAME-%SNAP_TYPE_recent

PRE_APP_UNQUIESCE_CMD01="C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s %SNAME-%SNAP_TYPE_recent -D F

 

The above assumes that there is already a snapshot that exists with the _recent naming.

This would DELETE the snapshot before quiescing Domino.

Snap Creator could continue doing its thing, including taking a snapshot.

You would then take an additional snapshot before unqiuescing Domino.

 

I tested this in my environment and can confirm that it works.

From my config:

PRE_APP_UNQUIESCE_CMD01="C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s %SNAME-%SNAP_TYPE_recent -D F

PRE_APP_QUIESCE_CMD01="C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap delete -d F -s %SNAME-%SNAP_TYPE_recent

 

From the logs:

 

########## PRE APPLICATION QUIESCE COMMANDS ##########

[2013-04-30 13:21:50,427] INFO: [10.63.168.250:9090 (4.0.0.1)] Executing Pre application quiesce command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap delete -d F -s Tam40c-daily_recent] on 10.63.168.250

[2013-04-30 13:21:50,427] DEBUG: [10.63.168.250:9090 (4.0.0.1)] Executing command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap delete -d F -s Tam40c-daily_recent]

[2013-04-30 13:21:50,427] TRACE: Command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap delete -d F -s Tam40c-daily_recent] finished with

exit code: [0]

stdout: [The operation completed successfully.

]

stderr: []

[2013-04-30 13:21:50,427] DEBUG: [10.63.168.250:9090 (4.0.0.1)] Command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap delete -d F -s Tam40c-daily_recent] finished successfully with message

[The operation completed successfully.

]

[2013-04-30 13:21:50,427] INFO: Pre application quiesce completed successfully

########## PRE APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY  ##########

[2013-04-30 13:21:50,427] INFO: Application auto discovery is not enabled skipping .

[2013-04-30 13:21:50,427] INFO: Validate volume is not enabled skipping validate volume task.

########## Application Quiesce ##########

.

.

.

########## SNAPSHOT CREATE COMMANDS ##########

[2013-04-30 13:22:08,130] INFO: [10.63.168.250:9090 (4.0.0.1)] Executing Snapshot create  command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_20130430132147 -D F G H] on 10.63.168.250

[2013-04-30 13:22:08,130] DEBUG: [10.63.168.250:9090 (4.0.0.1)] Executing command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_20130430132147 -D F G H]

[2013-04-30 13:22:08,130] TRACE: Command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_20130430132147 -D F G H] finished with

exit code: [0]

stdout: [Initializing...TAMPICO : Checking policies

TAMPICO : Checking access control

TAMPICO : Preparing LUNs for Snapshot creation

TAMPICO : Ready to create snapshot copy

TAMPICO : Creating a Snapshot for the LUN

TAMPICO : Initiating snapmirror update for any source volumes.

The operation completed successfully.

]

stderr: []

[2013-04-30 13:22:08,130] DEBUG: [10.63.168.250:9090 (4.0.0.1)] Command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_20130430132147 -D F G H] finished successfully with message

[Initializing...TAMPICO : Checking policies

TAMPICO : Checking access control

TAMPICO : Preparing LUNs for Snapshot creation

TAMPICO : Ready to create snapshot copy

TAMPICO : Creating a Snapshot for the LUN

TAMPICO : Initiating snapmirror update for any source volumes.

The operation completed successfully.

]

[2013-04-30 13:22:08,130] INFO: Snapshot create  completed successfully

########## Snapshot copy create commands finished successfully ##########

.

.

.

########## PRE APPLICATION UNQUIESCE COMMANDS ##########

[2013-04-30 13:22:12,068] INFO: [10.63.168.250:9090 (4.0.0.1)] Executing Pre unapplication quiesce command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_recent -D F] on 10.63.168.250

[2013-04-30 13:22:12,068] DEBUG: [10.63.168.250:9090 (4.0.0.1)] Executing command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_recent -D F]

[2013-04-30 13:22:12,068] TRACE: Command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_recent -D F] finished with

exit code: [0]

stdout: [Initializing...TAMPICO : Checking policies

TAMPICO : Checking access control

TAMPICO : Preparing LUNs for Snapshot creation

TAMPICO : Ready to create snapshot copy

TAMPICO : Creating a Snapshot for the LUN

TAMPICO : Initiating snapmirror update for any source volumes.

The operation completed successfully.

]

stderr: []

[2013-04-30 13:22:12,068] DEBUG: [10.63.168.250:9090 (4.0.0.1)] Command ["C:\Program Files\NetApp\SnapDrive\sdcli.exe" snap create -s Tam40c-daily_recent -D F] finished successfully with message

[Initializing...TAMPICO : Checking policies

TAMPICO : Checking access control

TAMPICO : Preparing LUNs for Snapshot creation

TAMPICO : Ready to create snapshot copy

TAMPICO : Creating a Snapshot for the LUN

TAMPICO : Initiating snapmirror update for any source volumes.

The operation completed successfully.

]

[2013-04-30 13:22:12,068] INFO: Pre unapplication quiesce completed successfully

########## PRE APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY  ##########

########## Application Unquiesce ##########

.

.

.

########## Snap Creator Framework 4.0.0 finished successfully ##########

 

Looking at the storage I see this:

 

Name                                   Date Time

Tam40c-daily_recent                04/30/2013 13:17:02

Tam40c-daily-20130430132147        04/30/2013 13:16:58

Tam40c-daily-20130430131518        04/30/2013 13:10:30

 

Note that the _recent snapshot was taken a few seconds after the initial snapshot, but it was taken while the application is still quiesced.

 

Feel free to let me know if you have any additional questions.

Comments

Thanks for the detailed explanation John.  I am the one who recently posted the feature request in the SnapCreator idea section. I'm a Netapp Partner and we have a handful of customers still running Domino that would certainly benefit from this functionality.  I understand it is a lot of work to fix and you have to weigh the cost benefit.  I really appreciate you taking the time to explain the reason why it's not in place today and for providing a viable work-around.  Some of our customers are not as open to work-arounds as others, but we will certainly employ this method where we can.

Keep up the good work.  Love the SnapCreator product!

Thanks,

Chris

Frequent Contributor

Chris,

Happy to provide the write-up and possible work around. 

I know the workaround isn't perfect, but until we can get adding the _recent option scoped and scheduled it is the best that I can offer.

In the meantime if you have other comments or questions please feel free to reach out and we'll do what we can to help.

Thanks for posting the idea and prompting the blog entry!

John