Data Backup and Recovery

SnapCreator and Qtree SnapMirror used as "SnapVault" backups

uptimenow
4,753 Views

Hi,

I am implementing SC at a customer. Currently, he is using QSM (no SV licenses), and he is

using QSM as a SnapVault alternative: eg. He is taking SMSQL/SME backups on the first line

with a retention of 7 backups, he is replicating the qtrees that contain the LUNs using QSM,

based on the snapshots created by SMSQL/SME. Finally, on the destination, he is taking

snapshots with a retention of 31 snapshots.

I have already implemented SC+SV at another customer, and I thought the SC+QSM integration

would be very similar, i.e. I was hoping that SC would offer me the possibility to define the

number of snapshots I would like to keep on the QSM destinations (for SV, this is can be

achieved by setting the NTAP_SNAPVAULT_RETENTIONS parameter). I was hoping that

there would be a NTAP_SNAPMIRROR_RETENTIONS parameter that would be used in QSM

scenario's.

There isn't such a parameter, and what basically happens, is: SC performs a QSM update

of all the relations, but that's basically it. The first time, SC also created a snapshot on

the QSM destinations: "sc_smsql-daily_recent". On subsequent runs of the backup, it tries

to create that same snapshot again (which causes a console error on the filer, saying the

snapshot already exists).

At the moment, QSM support in SC is fairly useless in its current incarnation, so my first

question is:

(1) Are there plans to change this and give some SV-like behavior for QSM replications ?

OK, with this in mind, I thought about implementing a workaround. This is actually possible,

because I can run additional commands via this parameter:

POST_NTAP_DATA_TRANSFER_CMDxx=

What is even more interesting is the fact the I can re-use environment variables of configuration

parameters that are set in the configuration, for instance %VOLUMES will get me the list of

volumes for which to replicate the qtrees.

One question - question number two - I have about this, is:

(2) "POST_NTAP_DATA_TRANSFER_CMDxx" can be set in the configuration file, but is not

shown in the Web GUI (SC 3.6.0c). Application quiesce, unquiesce, pre-exit and snapshot

create commands are visible from the GUI. Is there any chance of making the

"POST_NTAP_DATA_TRANSFER_CMDxx" editable via the Web GUI in a future release ?

Actually this goes for all the "POST" commands that can be added to the configuration

files.

Lastly, something I tried out today and that amazed me (SnapCreator is really cool

for this sort of creative tweaking !). Out of curiosity, I put the following in the configuration

file:

NTAP_SNAPMIRROR_RETENTIONS=daily:20

This is a non-existing and non-documented parameter, yet, when running a backups with this

parameter, SC did not throw an error message, and I was even able to access the

contents of the parameter during execution via the environment variable

%NTAP_SNAPMIRROR_RETENTIONS

My question is:

(3) are users allowed to add their own "custom" variables to a SC configuration ?

Don't get me wrong, I like this, because I can make use of this. But it does open a

door to a whole mess of problems if one day NetApp decides to use the same

parameter in an official SC release. What would be the recommended practice here ?

Finally, I am still stuck with the fact that I will have to write a "snapshot-rotate" script

that performs snapshot rotation on the QSM destinations. While it's not too

difficult to use rsh or plink.exe/ssh to launch commands on the filer, I am

wondering if there is actually a way to access the filer credentials that are stored

in the configuration file (I am using hashed passwords though), ideally I

would like to know if there is some easy-access API available through SC

to achieve this. I don't think it's possible, but it's worth a shot.

I would like to know if my demand for better support of QSM setups would be

considered as a feature request for a future release ...

Best regards,

Filip

1 ACCEPTED SOLUTION

ktenzer
4,753 Views

Hi Filip,

Let me first answer overall question of how to do QSM like SnapVault.

First you need two configs. One will be used to do QSM update and wait for QSM update to finish, this requires a specific option. The second config will be setup for secondary and will just delete snapshots. Finally you can call the second config from the first config.

NTAP_CLONE_SECONDARY=Y should be set in first config, this forces QSM to wait until complete, checking for errors.

In second config just point it at secondary storage volumes and set retention you want NTAP_SNAPSHOT_RETENTIONS

NTAP_SNAPSHOT_DISABLE=Y should be set in second config, this will disable snapshot creation, menaing if you set nothing else and use --action snap all it will do is delete.

In addition both configs should share same value for SNAME otherwise second config wont find snapshots.

You can call second config in wrapper script where you run one config then next or from first config using NTAP_POST_CMDs.

Ok this is this perfect, no, but hey it is a solution and should work. Keep in mind I havent really tried this but it sounds doable, SC has proven to do far more complex things

Ok now on to your specific questions:

(1) Are there plans to change this and give some SV-like behavior for QSM replications ?

No because 1) PM 2) It should work or you should be able to make it work as defined above 3) It isnt requested much by customers so if we get enough requests we would certainly do it in future but that hasnt happened.

 

(2) "POST_NTAP_DATA_TRANSFER_CMDxx" can be set in the configuration file, but is not

shown in the Web GUI (SC 3.6.0c). Application quiesce, unquiesce, pre-exit and snapshot

create commands are visible from the GUI. Is there any chance of making the

"POST_NTAP_DATA_TRANSFER_CMDxx" editable via the Web GUI in a future release ?

Yes over time we have been adding more parameters to UI but there are so many pre / post commands so we have avoided it due to clutter reasons until we could improve UI. In 4.0 we have a new implementation of UI will will eventually allow for this so its coming

Actually this goes for all the "POST" commands that can be added to the configuration

files.

Lastly, something I tried out today and that amazed me (SnapCreator is really cool

for this sort of creative tweaking !). Out of curiosity, I put the following in the configuration

file:

NTAP_SNAPMIRROR_RETENTIONS=daily:20

This is a non-existing and non-documented parameter, yet, when running a backups with this

parameter, SC did not throw an error message, and I was even able to access the

contents of the parameter during execution via the environment variable

%NTAP_SNAPMIRROR_RETENTIONS

Yes this is working as intended We expose anything set in config to ENV so you can do a printenv and see the key / value pairs. Even more so we expose it to scAgent so any script running even on scAgent will inherit them...furthermore we also expose it to the plugins so those writing plugins have easy way to collect data through config in order to make logic decisions. A plugin is also a way to do anything you want and have plugin drive SC workflows.

My question is:

(3) are users allowed to add their own "custom" variables to a SC configuration ?

Yes

Don't get me wrong, I like this, because I can make use of this. But it does open a

door to a whole mess of problems if one day NetApp decides to use the same

parameter in an official SC release. What would be the recommended practice here ?

Well as long as parameter name is unique you are fine. Almost all parameters start with NTAP, but if you are specific about parameter you will never have issue. For example say you wanted to store a user name you could do this: USER=joe. This is not descriptive and could create a problem maybe...still very low chance. However CUSTOM_ORACLE_USER=joe is much more descriptive.

I would recommend strongly using CUSTOM in front of parameter. That is a placeholder we wont use. In future maybe we could limit parameters so that if you add one it must start with CUSTOM, but we didnt want to limit it. Basically you better have a really good reason to limit flexibility so if someone can come up with one I will certainly entertain it This is very different to the approach of other products where flexibility is often totally gimped due to support / test concerns, but that is not the SC way.

Please let me know about idea w/two configs and if it works. There was a bug in SC 3.5 with cascading SC commands so calling SC from another SC config, an endless loop condition which could occur if you enabled ASUP errors so turn that off (it is off by default). This is fixed in 3.6 but if you run into just wanted to make you aware. SC 3.6 will be available in next few weeks and community release is already available on www.snapcreator.com.

Happy SnapCreating!

Keith

View solution in original post

4 REPLIES 4

TAYLORRIGGAN
4,753 Views

Hey Filip,

You may want to try integrating SnapCreator with Protection Manager.  SnapCreator can interface with a Dataset in Protection and Manager and Protection Manager can control the QSM relationship, along with control the snapshot retention on the destination volume.

Here's a link to a video about this integration:  https://communities.netapp.com/videos/1998

Hope that helps.

uptimenow
4,753 Views

Hi Taylor,

You are right, and I did think of that too, but I would like to avoid Protection Manager if at all possible (the customer would like to avoid it too). I like the SnapCreator approach better. Otherwise we could have implemented a SMSQL/SME+PM solution without the need for SC altogether. However, there are other applications in this customer's setup that don't integrate with PM, and that we can backup with PM.

Still, thanks for the suggestion!

Best regards,

Filip

ktenzer
4,754 Views

Hi Filip,

Let me first answer overall question of how to do QSM like SnapVault.

First you need two configs. One will be used to do QSM update and wait for QSM update to finish, this requires a specific option. The second config will be setup for secondary and will just delete snapshots. Finally you can call the second config from the first config.

NTAP_CLONE_SECONDARY=Y should be set in first config, this forces QSM to wait until complete, checking for errors.

In second config just point it at secondary storage volumes and set retention you want NTAP_SNAPSHOT_RETENTIONS

NTAP_SNAPSHOT_DISABLE=Y should be set in second config, this will disable snapshot creation, menaing if you set nothing else and use --action snap all it will do is delete.

In addition both configs should share same value for SNAME otherwise second config wont find snapshots.

You can call second config in wrapper script where you run one config then next or from first config using NTAP_POST_CMDs.

Ok this is this perfect, no, but hey it is a solution and should work. Keep in mind I havent really tried this but it sounds doable, SC has proven to do far more complex things

Ok now on to your specific questions:

(1) Are there plans to change this and give some SV-like behavior for QSM replications ?

No because 1) PM 2) It should work or you should be able to make it work as defined above 3) It isnt requested much by customers so if we get enough requests we would certainly do it in future but that hasnt happened.

 

(2) "POST_NTAP_DATA_TRANSFER_CMDxx" can be set in the configuration file, but is not

shown in the Web GUI (SC 3.6.0c). Application quiesce, unquiesce, pre-exit and snapshot

create commands are visible from the GUI. Is there any chance of making the

"POST_NTAP_DATA_TRANSFER_CMDxx" editable via the Web GUI in a future release ?

Yes over time we have been adding more parameters to UI but there are so many pre / post commands so we have avoided it due to clutter reasons until we could improve UI. In 4.0 we have a new implementation of UI will will eventually allow for this so its coming

Actually this goes for all the "POST" commands that can be added to the configuration

files.

Lastly, something I tried out today and that amazed me (SnapCreator is really cool

for this sort of creative tweaking !). Out of curiosity, I put the following in the configuration

file:

NTAP_SNAPMIRROR_RETENTIONS=daily:20

This is a non-existing and non-documented parameter, yet, when running a backups with this

parameter, SC did not throw an error message, and I was even able to access the

contents of the parameter during execution via the environment variable

%NTAP_SNAPMIRROR_RETENTIONS

Yes this is working as intended We expose anything set in config to ENV so you can do a printenv and see the key / value pairs. Even more so we expose it to scAgent so any script running even on scAgent will inherit them...furthermore we also expose it to the plugins so those writing plugins have easy way to collect data through config in order to make logic decisions. A plugin is also a way to do anything you want and have plugin drive SC workflows.

My question is:

(3) are users allowed to add their own "custom" variables to a SC configuration ?

Yes

Don't get me wrong, I like this, because I can make use of this. But it does open a

door to a whole mess of problems if one day NetApp decides to use the same

parameter in an official SC release. What would be the recommended practice here ?

Well as long as parameter name is unique you are fine. Almost all parameters start with NTAP, but if you are specific about parameter you will never have issue. For example say you wanted to store a user name you could do this: USER=joe. This is not descriptive and could create a problem maybe...still very low chance. However CUSTOM_ORACLE_USER=joe is much more descriptive.

I would recommend strongly using CUSTOM in front of parameter. That is a placeholder we wont use. In future maybe we could limit parameters so that if you add one it must start with CUSTOM, but we didnt want to limit it. Basically you better have a really good reason to limit flexibility so if someone can come up with one I will certainly entertain it This is very different to the approach of other products where flexibility is often totally gimped due to support / test concerns, but that is not the SC way.

Please let me know about idea w/two configs and if it works. There was a bug in SC 3.5 with cascading SC commands so calling SC from another SC config, an endless loop condition which could occur if you enabled ASUP errors so turn that off (it is off by default). This is fixed in 3.6 but if you run into just wanted to make you aware. SC 3.6 will be available in next few weeks and community release is already available on www.snapcreator.com.

Happy SnapCreating!

Keith

uptimenow
4,753 Views

Thanks for your very fast reply, Keith !

Let me first answer overall question of how to do QSM like SnapVault.

...

Ok this is this perfect, no, but hey it is a solution and should work. Keep in mind I havent really tried this but it sounds doable, SC has proven to do far more complex things

OK, I understand what you're pointing at. It's a bit of a creative hack too, in fact 🙂

I will discuss it with the customer during my next visit (which is by the end of the month), and we

will decide whether we'll work with two config files, or write a little script to trigger the snapshot

rotation. If we decide to write a little script, I'll attach it to this thread.

No because 1) PM 2) It should work or you should be able to make it work as defined above 3) It isnt requested much by customers so if we get enough requests we would certainly do it in future but that hasnt happened.

OK, fair enough.

By the way, I don't know if this is true for all SnapCreator users, but the customers I am currently implementing SnapCreator at want to avoid PM and are quite happy that SC can take care of  SV/SM updates in a clean and standard way.

Yes over time we have been adding more parameters to UI but there are so many pre / post commands so we have avoided it due to clutter reasons until we could improve UI. In 4.0 we have a new implementation of UI will will eventually allow for this so its coming

OK, cool.

Yes this is working as intended We expose anything set in config to ENV so you can do a printenv and see the key / value pairs. Even more so we expose it to scAgent so any script running even on scAgent will inherit them...furthermore we also expose it to the plugins so those writing plugins have easy way to collect data through config in order to make logic decisions. A plugin is also a way to do anything you want and have plugin drive SC workflows.

That is really cool. I was wondering if it is also possible to send the agent logfiles back to the SC server after a job ? It would be really nice to look into those logfiles from the SC server GUI.

I would recommend strongly using CUSTOM in front of parameter.


Makes a lot of sense, thanks !

Please let me know about idea w/two configs and if it works. There was a bug in SC 3.5 with cascading SC commands so calling SC from another SC config, an endless loop condition which could occur if you enabled ASUP errors so turn that off (it is off by default). This is fixed in 3.6 but if you run into just wanted to make you aware. SC 3.6 will be available in next few weeks and community release is already available on www.snapcreator.com.

I will, but it'll still take a week or two before I get a chance to test out your suggestion.

Best regards & thanks again for all the help,

Filip

Public