Exchange backup with vault update not working

[MIKKELKJAER]

Hi

I'm currently trying to configure Exchange 2010 backup on Clustered ONTAP 8.2. The backup should be cascading the following way:

Primary => SnapVault => SnapMirror

This configuration is not supported by SME 6.0 and doesn't appear to be supported by Snap Creator 4.1.0.c. Instead I'm trying to archive the first step:

Primary => SnapVault

I'm then hoping to perform the SnapMirror update manually later on (perhaps from Snap Creator).

However, while trying to perform the initial primary and vault backup I get the following error from Snap Creator: "Unable to find SnapVault relationships".

I have configured both SnapVault and SnapMirror volumes and relationships using Workflow Automation so complete baselines have been completed.

Question: Is the above configuration supported? And if yes - what am I doing wrong? :-?

Best regards,

Mikkel Kjaer

[MIKKELKJAER]

An additional note. I'm allowed to save the configuration and get the following error when trying to run the backup:

########## Running storage discovery ##########

[2013-12-05 13:51:35,638] INFO: STORAGE-02113: Retrieving SnapVault relationships

[2013-12-05 13:51:35,638] INFO: STORAGE-02114: Retrieving SnapVault relationships on controller [sv01] finished successfully

[2013-12-05 13:51:35,638] ERROR: SCF-00019: SnapVault relationship for [sv01]:[cmexch02_data02_vault001] does not exist, Exiting!

sv01 is the SnapVault Vserver and cmexch02_data02_vault001 is one of the destination volumes.

Here is more information about the configured SnapVault destination:

nac-sv::> volume show -vserver sv01

Vserver   Volume       Aggregate    State      Type       Size  Available Used%

--------- ------------ ------------ ---------- ---- ---------- ---------- -----

sv01      cmexch02_data02_vault001

                       data1        online     DP         10GB     7.98GB   20%

nac-sv::> snapmirror show -vserver sv01 -destination-volume cmexch02_data02_vault001

                                                                       Progress

Source            Destination  Mirror  Relationship  Total             Last

Path        Type  Path         State   Status        Progress  Healthy Updated

----------- ---- ------------ ------- -------------- --------- ------- --------

prod:cmexch02_data02

            XDP  sv01:cmexch02_data02_vault001

                              Snapmirrored

                                      Idle           -         true    -

[spinks]

Would you mind attaching the debug log or copying/pasting the section in which the error occurred?

This error is given to us by ONTAP, so ONTAP is reporting that the relationship does not exist.

Be sure to check capitalization as ONTAP is case sensitive and be sure that your DNS resolution is resolving to the correct hosts.

For example, if the Snap Creator server can't resolve sv1 the command will fail.

These are simple things, but I've seen them trip people up in the past...

Also, try running the SnapMirror update command through the controller CLI and see what response you get.

Thanks,

John

[MIKKELKJAER]

Hi John

This is the first error that occur in the backup debug log:

########## Running storage discovery ##########

[2013-12-05 13:51:35,638] INFO: STORAGE-02113: Retrieving SnapVault relationships

[2013-12-05 13:51:35,638] INFO: STORAGE-02114: Retrieving SnapVault relationships on controller [sv01] finished successfully

[2013-12-05 13:51:35,638] ERROR: SCF-00019: SnapVault relationship for [sv01]:[cmexch02_data02_vault001] does not exist, Exiting!

Both primary and vault Vservers are resolvable from the Snap Creator server host (through DNS). Lower case are used everywhere. Vserver names and volumes have been specified using Snap Creator GUI so they should match correct upper/lower case.

Running a manual snapmirror update on the vault site returns successfully (nothing is transferred since there are no new snapshots on the primary volume):

nac-sv::> snapmirror update -destination-path sv01:cmexch02_data02_vault001

Operation is queued: snapmirror update of destination sv01:cmexch02_data02_vault001.

nac-sv::> log show

  (event log show)

Time                Node             Severity      Event

------------------- ---------------- ------------- ---------------------------

12/5/2013 14:44:24  svnode           INFORMATIONAL sm.vlt.xfer.no.new.snap: SnapVault update did not find any new Snapshot copies to transfer from the source volume 'prod:cmexch02_data02'.

Seems like SnapMirror (vault) update is working but the relationship is not identified or recognized by SC.

Best regards,

Mikkel

[MIKKELKJAER]

Hi John

I think it got the vault update partly working.

There were some configuration errors regarding volume/Vserver specification. Also I had to specify all Vserver credentials with short hostname.

Now the Exchange primary snapshots are being performed and a SnapVault update is triggered according to the log.

However... The vault is being updated with an old snapshot - not the latest snapshot created by SME.

Shouldn't the vault update use the SME snapshot created during the backup?

Best regards,

Mikkel

[spinks]

Mikkel,

Glad you got part the first part.

To clarify - using the fully qualified name for your storage caused errors, but using the short named worked ok?

I suspect this is an ONTAP limitation, but I'll try to replicate the behavior.

As for the second part I see in the logs:

[2013-12-11 13:25:24,515] INFO: External Snapshot copy exchsnap__cmexch02_12-04-2013_14.17.26__daily will be used for prod:cmexch02_log01

So your action on 12/11 is using a Snapshot from 12/4.

Can you confirm if a Snapshot was taken on 12/11 on the volume mentioned above?

I'm going to ping someone on our developer team who knows more about this plugin than I do - I'm not certain about this behavior.

Thanks,

John

[MIKKELKJAER]

Hi John

Regarding the short/long name. I did add some entries to the local host files on SC and SME server to make sure I was not facing host resolution issues.

In addition to this there were some configuration errors (I'm not 100% into the "Snap Creator logic" when it comes to specifying volume entities in different parts of the SC configuration).

New SME snapshots were created successfully on the primary volume:

nac1::> snapshot show -vserver prod -volume cmexch02_log01

                                                                   ---Blocks---

Vserver  Volume  Snapshot                        State        Size Total% Used%

-------- ------- ------------------------------- -------- -------- ------ -----

prod     cmexch02_log01

...

                 exchsnap__cmexch02_12-11-2013_12.52.03__daily

                                                 valid      4.73MB     0%    2%

                 eloginfo__cmexch02_12-11-2013_12.52.03__daily

                                                 valid       544KB     0%    0%

                 exchsnap__cmexch02_12-11-2013_12.57.00__daily

                                                 valid      4.82MB     0%    3%

                 eloginfo__cmexch02_12-11-2013_12.57.00__daily

                                                 valid       544KB     0%    0%

[spinks]

Mikkel,

I pinged someone else and they gave me back this information:

Be sure to use SC 4.0 or higher and the external snapshot and external snapshot regex options.

This will allow SC to use an existing snapshot and use the latest which matches the regex.

I asked for a config file that could be used as an example, but I haven't been able to get one.

It sounds like the regex options need to be updated to be sure that the newest Snapshor copy with the proper name is in use.

If I can get better info I'll try and update the thread.

Thanks!

John

[MIKKELKJAER]

Hi John

The SC version used is the latest community version (4.1c). I tried the backup with the following Snapshot options in the config file:

NTAP_USE_EXTERNAL_SNAPSHOT=Y

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*__daily

I would expect this to match the daily backup performed by SME.

When I run the backup, however, the first (oldest) snapshot is selected. Furthermore it seems like the snapshot "selection" occurs prior to performing the actual SME/Exchange backup. Could something be configured incorrectly?

I'll attach screenshot and logs from the last backup attempt.

Best regards,

Mikkel

[ktenzer]

Yeah the snapshot list happens when SC does storage discovery. You would want to snapshot to already exist but you could maybe change the workflow xml file and run discovery after the snapshot is taken or you could run SME and make sure snapshot is there before calling config which is supposed to do something with snapshot.

Under scServer/engine/etc/workflow are the XMLs. You can edit backup.xml and add a second storageDiscovery task after the task which is calling SME

<step task="storageDiscovery" configs="#APP_VM" abortStep="appCleanup">

</step>

I have never tried this so I have no idea if this would work but it is worth a try.

As far as the oldest snapshot being chosen, I am not sure about this, that could be a bug. We need to test if that is actually happening.

Regards,

Keith

[MIKKELKJAER]

Hi Keith

Thanks for the tip regarding changing the flow order on discovery and backups.

However, changing the order of events will not resolve the issue completely. Currently the oldest (first) snapshot is being matched for the vault update.

Are there additional parameters to configure in order to make Snap Screator pick the latest (newest) snapshot for the replication?

Best regards,

Mikkel

[ktenzer]

Nope the only parameter besides turning on external snapshots is NTAP_EXTERNAL_SNAPSHOT_REGEX, maybe try changing your regex to NTAP_EXTERNAL_SNAPSHOT_REGEX=__daily. It seems like SC is not matching the correct snapshots.

Keith

[ktenzer]

In addition if you still have problems please send another snap list from controller and a debug log to go along with it.

Keith

[MIKKELKJAER]

Hi Keith

I set the variable to the following:

NTAP_EXTERNAL_SNAPSHOT_REGEX=__daily

(pattern used to be =.*__daily) This causes the backup to fail. It looks like the snapshot is not being matched.

Here is a list of snapshots on the controller:

nac1::> snapshot show -vserver prod -volume cmexch02_data01

                                                                   ---Blocks---

Vserver  Volume  Snapshot                        State        Size Total% Used%

-------- ------- ------------------------------- -------- -------- ------ -----

prod     cmexch02_data01

                 exchsnap__cmexch02_12-04-2013_14.17.26__daily

                                                 valid      4.46MB     0%    2%

                 snapmirror.8147c403-4b7e-11e3-870e-123478563412_2147484692.2013-12-05_103417

                                                 valid      4.27MB     0%    2%

                 exchsnap__cmexch02_12-25-2013_14.17.55__daily

                                                 valid      1.19MB     0%    1%

                 exchsnap__cmexch02_12-26-2013_14.17.55__daily

                                                 valid      1.16MB     0%    1%

                 exchsnap__cmexch02_12-27-2013_14.17.55__daily

                                                 valid      1.14MB     0%    1%

                 exchsnap__cmexch02_12-28-2013_14.17.55__daily

                                                 valid      1.16MB     0%    1%

                 exchsnap__cmexch02_12-29-2013_14.17.55__daily

                                                 valid      1.08MB     0%    1%

                 exchsnap__cmexch02_12-30-2013_14.17.58__daily

                                                 valid      1.13MB     0%    1%

                 exchsnap__cmexch02_12-31-2013_14.17.54__daily

                                                 valid      1020KB     0%    1%

                 exchsnap__cmexch02_01-01-2014_14.17.55__daily

                                                 valid      1.05MB     0%    1%

10 entries were displayed.

[patro]

Hi ,

I have looked into this issue.

The actual root issue is the snapvault update is updating the old snapshot.

This was a BUG in 4.1.0c and 4.1.0 GA release.

This issue is fixed in 4.1p1 patch release .For details please refer the BURT 797655

http://burtweb-prd.eng.netapp.com/burt/burt-bin/start?burt-id=+797655+&view_button=view&button=view&google_text=&sel_report=&report=&report_user=patro...

The above fix also solves the issue for Extenal snapshot as well.I have verified this in 4.1p1 build  with following pattern :- NTAP_EXTERNAL_SNAPSHOT_REGEX=.*-daily.*

NOW Snapvault update happens for the latest snapshot as per the pattern defined in the parameter above.

Regards

hari