How to utilize SnapShots and SnapMirror with Oracle Database?

[LSOTECHUSER]

We will be migrating our database (Oracle 11g RAC) from an older Equallogic storage system to a NetApp FAS 2240 in the coming months.  Currently the database is using ASM and is connected via iSCSI.  When we move to the NetApp, at least initially we will use iSCSI and continue to use ASM. We would like to take advantage of Snapshots and Snapmirror when the database is moved to the NetApp.  We will not be using SnapManager for Oracle at this time.

I am not quite sure how the NetApp SnapShot process integrates with the Oracle and what pieces are required to make this work.

How are Snapshots and Snapmirror handled so that the database is backed up properly?  In particularly, how is the backup scheduled so that the database is put into hot or online backup mode when the snapshot and snapmirror runs?  Is everything handled from the Oracle host side via a script that can make the database ready and then trigger the snapshot on NetApp or do you need to coordinate the snapshot on NetApp separately with Oracle?

thanks

[oommen]

Hello - wondering is there a reason not to use SMO?

[LSOTECHUSER]

Well the perceived complexity is one reason.  For this migration we are simply wanting to get the Database on to the NetApp, but would still like to take advantage of basic snapshot and more importantly Snapmirror so we can have a copy of the database offsite.

In looking into SMO it sounds like it would require quite a few components to work and I don't think we have the time to bring it all together properly.  The SMO documentation I have read also mentions we need Protection Manager in order to use which we don't have (I cant figure out if this is even a product NetApp anymore?).  Maybe I am wrong in my assumption as it is a chore to keep track of all of NetApp's separate applications and technologies.

[nkarthik]

SMO with Protection Manager integration is not mandatory for snapmirror or snapvault update to protect the backup. You can try with backup with post script in SMO.

Regards,

Karthikeyan.N

[billshaffer]

Without SMO, there really isn't much "integration."  It can all be scripted for coordination, either on the DB server or on a management server.  You can either use SnapManager for windows/unix to run the snaps, or you can just have an ssh key from the server to the controllers to issue the snap commands.  Basically, you have a script that puts the DB in hot standby mode (or similar), then you connect to the filer and take the snapshot, then you bring the DB out of hot standby.  Depending on what your goals are, SnapMirror may be an independant topic - technically you can have the DB in hot standby for the duration of the snapmirror update, but many places don't want to spend that much time in hot standby.  As long as your snapshots are consistent, SnapMirror replication can happen while the DB is active - your target volume will be inconsistent, but the target snaps will all be fine.

Hope that helps...

Bill

[LSOTECHUSER]

Yes Bill that does help, thanks.  A few questions though:

"You can either use SnapManager for windows/unix to run the snaps" 

You are referring to SMO when you say SnapManager correct?

Depending on what your goals are, SnapMirror may be an independant topic

At least initially, my goal with SnapMirror is to get a consistent copy of the data base to our off site location to satisfy disaster recovery requirements.

technically you can have the DB in hot standby for the duration of the snapmirror update, but many places don't want to spend that much time in hot standby. 

So I understand correctly, when a snapmirror job is initiated a snapshot of the volume (called volume_name_snapmirror. etc) is created and then this snapshot is then copied correct?  It sounds like a script would require the database to be in Hot standby until the snapmirror update is complete?

As long as your snapshots are consistent, SnapMirror replication can happen while the DB is active - your target volume will be inconsistent, but the target snaps will all be fine.

If I schedule a script to run at 12:00 AM each day that puts the database into hot standby and then sends commands to the NetApp to create a snapshot of the volumes; I can then simply do a snapmirror update after this "consistent" snapshot is created.  The snapmirror volume at the DR site however will not be consistent, but the SnapShot that was scheduled at 12:00AM would be.  Is this correct?  This would mean I would have to have the database in hot standby for the duration of the snapmirror update correct?

[billshaffer · Accepted solution]

There are many snapmanager products - the basic ones are snapmanager for windows/unix.  All these do is intall some commands through which you can manage the snapshots on the controller - for example, snap list, snap create, snap delete, etc.  The connection is over one of the API channels.  The other snapmanager products (for Oracle, Exchange, SharePoint, etc.)  utilize the basic snapmanager products for talking to the controllers, but they also talk to the application in question, and will handle all the scheduleing and integration.

Remember that, in a snapmirror relationship, you're copying the source volume and snapshots.  What you say about updating a snapmirror relationship creating a snap that is copied to the destination is technically correct, but that process is part of the internal workings, and I find it better just to focus on the volume and non-snapmirror snapshot data.  So if you put your DB in hot standby, then issue a snapmirror update, wait for it to finish, then take your DB out of hot standby, you've got a consistent copy of your DB at DR that you could snapmirror back if needed.

If, however, you put your DB in hot standby, take a snapshot, take your DB out of hot standby, and THEN inititate the snapmirror update - which you don't need to wait for to finish - then your destination _volume_ is inconsistent from a DB perspective, because the DB was active when it was copied.  The snapshot you took when the DB was in hot standby, though, also gets copied - and that is consistent.  Instead of just reversing the direction of the snapmirror, you'd have to break the snapmirror, restore the volume to the desired snapshot on the destination, then mirror it back.

One caveat would be that a snapmirror reversal (as in my first scenario) would only copy incremental changes, while the second scenario would be a full copy - irrelevant in the event of a real disaster,but possibly impactful when faced with corruption.  Another benefit of doing just the snapshots would be that you could keep multiple consistent copies around as well.

Make sense?

Bill

[moglesby]

billshaffer wrote:
There are many snapmanager products - the basic ones are snapmanager for windows/unix.  All these do is intall some commands through which you can manage the snapshots on the controller - for example, snap list, snap create, snap delete, etc.  The connection is over one of the API channels.  

Bill,

There is no product called "SnapManager for Windows/UNIX."  It sounds like you are referring to "SnapDrive for Windows/UNIX" here.  That is probably the cause of some of LSOTECHUSER's confusion.

-Mike

[billshaffer]

Aha!  You are correct Mike.  I apologize for the confusion!

Bill

[LSOTECHUSER]

Thank you all for the good info.  It is making sense, however, as there are several pieces involved I want to make sure I have them correct.

Can someone verify these steps as correct:

In my scenario I would run any scheduled scripts from the Oracle database hosts.

1. To do so, I understand I need to be able to issue commands to the NetApp, such as 'snap create' from within my script.
   • To leverage these commands (ie. Snap create , snap list, snap delete, etc)  I need to install a NetApp component on the Oracle database server.
     • Question, do to some confusion with SnapManger/Snapdrive - What is this NetApp component that provides these commands and should be installed on the oracle database server: SnapDrive? or something else?  
2. After installing the piece that provides the NetApp commands, I understand that I would simply then create a script that would perform the snapshot or the snapmirror update.
   • This script would simply run RMAN commands to put database into hot backup mode and then it would create the snapshot, then put database back into normal mode
   • I could schedule the script using crontab        

• Question, are there any example scripts out there that are available that create Oracle Snapshot?

Lastly, as a related question, I am curious what you guys would recommend as far as volume setup?  Is it better to have 1 volume that  contains multiple LUNS (Data, Flash Recover, temp, etc) or is it better to have only 1 LUN per volume, thereby having multiple volumes?

I ask this as having 1 volume would be quite simple for consistency sake regarding snapshots and snapmirror.  However, maybe this is not best practice.  Also it sounds like I would then need more NetApp pieces to make sure snapshots are consistent when having multiple volumes.  I would imagine the main downfall with 1 volume containing multiple LUNS is that if there are multiple databases and you only need to recover 1; you would have to recover all of them. 

Thanks again for the assistance

[billshaffer]

The NetApp component is SnapDrive, but you don't NEED to install it.  SnapDrive will let you take and manage your snapshots from the server side, but it also opens up a lot of manageability options from the server (creating/deleting/resizing luns, etc.)  This may be overkill for you - and SnapDrive is a licensed product.  I've had very good experience with keyed SSH commands inside of scripts - in fact, for the past 10+ years that is almost the ONLY way I've scripted against NetApp.

As for example scripts, are you looking for an example of the whole process?  Or just of creating the snapshot, as you said?  ..snap create oracle_vol oracle_snap...  I imagine you'd be able to find a decent skeleton out there somewhere.

There are a bunch of different schools of thought around luns per volume.  I've seen recommendations of a single lun per volume, but to me that seems like overkill - though it DOES give you the maximum management ability (if not maximum simplicity).  One volume for all luns that make up a DB is, as you say, easy for management - but you can also make the argument for splitting out pieces that you don't need for recovery - for example, do you need temp and flash recovery bits replicated (sorry, I'm not a DBA...)?  If not, put those in a different volume - of course, if the size of these is negligible, then it doesn't really matter and one volume per DB sounds better.  I would avoid putting multiple DBs in a volume - unless, or course, all the DBs MUST be consistent together, then you could make the argument to put then in the same volume for simplicity - or put the bits needing replication in the same volume, and the bits that don't together in a different volume...  So I think the answer to your question is "it depends..."

All that being said, having the data in many different volumes doesn't really add THAT much complexity - multiple "snap create" commands while the DB is in hot backup is all...

One caveat about SSH commands against the filer is that you don't get return codes - so typically after creating a snapshot, I'll do a snap list and verify that it's there...

Bill

[LSOTECHUSER]

 I've had very good experience with keyed SSH commands inside of scripts - in fact, for the past 10+ years that is almost the ONLY way I've scripted against NetApp.

This sounds like an interesting option.  This means that there are no extra components that need to be installed on the Oracle servers.

I will have to look into "keyed" SSH commands as I have not done this before.  If I did do SSH though wouldnt  the username and password for the NetApp filer end up being stored and readable in the Script?  Maybe this is where "keyed SSH" comes in?  Sorry my scripting knowledge with Linux is minimal.

As for example scripts, are you looking for an example of the whole process?  Or just of creating the snapshot, as you said?  ..snap create oracle_vol oracle_snap...  I imagine you'd be able to find a decent skeleton out there somewhere.

I guess any examples would be great, but I was kind of looking for a complete example.  I have searched for Oracle and Linux scripts, but have had minimal luck.  Most things I have seen already assume you have a very good handle on the Oracle and NetApp setup.  This is where I am trying to get to.  I have been working with a test environment for the last several days and I am getting more familiar, but I am not there yet.

If you happen to have any scripts that you could sanitize and provide that would be most helpful.  Maybe PM?

Again thanks for all of the help!

[billshaffer]

Keyed SSH means you don't need a password to log in, so having the password in a script is not a concern.  Obviously there ARE security implications doing it this way.

Unfortunately I am no longer in an environment where we do this sort of thing, so I don't have any scripts handy - sorry!.

Bill

[moglesby]

Hi LSOTECHUSER,

LSOTECHUSER wrote:
To do so, I understand I need to be able to issue commands to the NetApp, such as 'snap create' from within my script.
To leverage these commands (ie. Snap create , snap list, snap delete, etc)  I need to install a NetApp component on the Oracle database server.
Question, do to some confusion with SnapManger/Snapdrive - What is this NetApp component that provides these commands and should be installed on the oracle database server: SnapDrive? or something else?   

SnapDrive for Windows/UNIX is the tool that is used by SnapManager for Oracle to communicate with the storage controller.  Likewise, you could use it to issue commands to the storage controller. 

Another option is Snap Creator Framework.  It provides very basic integration with Oracle - it can handle backup, but some scripting is required for restore/recover. 

As others have said, if you want full integration, SnapManager for Oracle is your best option.

LSOTECHUSER wrote:
Lastly, as a related question, I am curious what you guys would recommend as far as volume setup?  Is it better to have 1 volume that  contains multiple LUNS (Data, Flash Recover, temp, etc) or is it better to have only 1 LUN per volume, thereby having multiple volumes?
I ask this as having 1 volume would be quite simple for consistency sake regarding snapshots and snapmirror.  However, maybe this is not best practice.  Also it sounds like I would then need more NetApp pieces to make sure snapshots are consistent when having multiple volumes.  I would imagine the main downfall with 1 volume containing multiple LUNS is that if there are multiple databases and you only need to recover 1; you would have to recover all of them.  

Have you taken a look at the "Best Practices for Oracle Databases on NetApp Storage" document (http://www.netapp.com/us/system/pdf-reader.aspx?pdfuri=tcm:10-60340-16&m=tr-3633.pdf)?  Hopefully this will help.

Let us know if you have any more questions!

Thanks!

-Mike

[LSOTECHUSER]

SnapDrive for Windows/UNIX is the tool that is used by SnapManager for Oracle to communicate with the storage controller.  Likewise, you could use it to issue commands to the storage controller.

I have installed SnapDrive on a test server and it appears that it is fairly straight forward.  I was under the assumption that I had to create and setup all of the iSCSI connections via SnapDrive as opposed to using the open-iscsi intiator, but this appears to not be the case.  I installed it on my test server that was already connected to the NetApp and it looks like SnapDrive can detect the volumes.  If this is correct it looks like I can install SnapDrive simply to get the SnapDrive commands which I can use in my script, correct?

Another option is Snap Creator Framework.  It provides very basic integration with Oracle - it can handle backup, but some scripting is required for restore/recover. 

I am looking into this also, but it appears that I would need to run this on another server.

Have you taken a look at the "Best Practices for Oracle Databases on NetApp Storage" document

I had looked into this some time ago, but I need to revisit now that I have a better understanding of things .Looking at this now, thank you for the link.

Thank you for the help, much appreciated!

[moglesby]

LSOTECHUSER wrote:
 I installed it on my test server that was already connected to the NetApp and it looks like SnapDrive can detect the volumes.  If this is correct it looks like I can install SnapDrive simply to get the SnapDrive commands which I can use in my script, correct?

Yes, that is correct.

LSOTECHUSER wrote:
Another option is Snap Creator Framework.  It provides very basic integration with Oracle - it can handle backup, but some scripting is required for restore/recover. 
I am looking into this also, but it appears that I would need to run this on another server.

It does not necessarily need to be run on another server.  You can run both the Snap Creator Framework server and agent on the same host.