Problem with NTAP_PM_UPDATE SC 3.4.0 and Protection Magnager of OnCommand 5.0

[f_duranti]

Hi, I'm setting up a new backup from SC 3.4.0 with Protection Manager integration but it seems that I've some problem with this. The backup is composed of 8 volumes snapmirrored on a secondary filer.

This is what I've done:

1) Create a Dataset with pmsetup (then I've renamed it for our naming convention)

2) Create a primary (NFS) and secondary provisioning policy

3) Create a protection policy (it just have the lag time and no schedule because I'll schedule from cron with SC)

4) Create a resource pool for primary and secondary storage

5) Assigned the original volume to the primary storage, assigned the provisioning policy, the protection policy and the resource pool

6) OnCommand created the secondary volume with my naming convention and initialized the snapmirror relationship

At this point I create a backup with snapcreator --action snap

I'm expecting that in the Dataset, selecting the Dataset, in the "Primary Data", Backup Version I should see the button to see backup of this dataset and the date of the last backup but There's nothing and after the snapshot policy lag time the dataset go in error.

Going on the provision tab on a volume and on the detele snapshot to see the snapshot of the volume I see them as normal and not as "Backup" as they should be.

Checking the log SC says that it has registered the backup correctly but it's not visible.

I've an older test configured that was working correctly but I cannot find what can be wrong.

This is part of the config file that is not working:

CONFIG_TYPE=STANDARD

USE_PROXY=N

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

SNAME=BCK

SNAP_TIMESTAMP_ONLY=N

VOLUMES=itnapfs20:DSR_Data,DSR_Log,DSR_Temp,DSR_Exe,DJR_Data,DJR_Log,DJR_Temp,DJR_Exe

NTAP_SNAPSHOT_RETENTIONS=daily:7

NTAP_USERS=itnapfs20:user/pw

NTAP_PWD_PROTECTION=Y

TRANSPORT=HTTPS

PORT=443

LOG_NUM=10

SC_TMP_DIR=

########################

### Snapshot Options ###

########################

NTAP_SNAPSHOT_RETENTION_AGE=7

SNAPDRIVE=N

SNAPDRIVE_DISCOVERY=N

NTAP_SNAPSHOT_DISABLE=N

NTAP_SNAPSHOT_NODELETE=Y

NTAP_CONSISTENCY_GROUP_SNAPSHOT=Y

NTAP_CONSISTENCY_GROUP_TIMEOUT=relaxed

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=Y

##########################

### SnapMirror Options ###

##########################

NTAP_SNAPMIRROR_UPDATE=N

NTAP_SNAPMIRROR_CASCADING_UPDATE=N

SNAPMIRROR_VOLUMES=

SNAPMIRROR_CASCADING_VOLUMES=

NTAP_SNAPMIRROR_WAIT=0

NTAP_SNAPMIRROR_USE_SNAPSHOT=N

NTAP_SNAPMIRROR_MAX_TRANSFER=

##################################

### Protection Manager Options ###

##################################

NTAP_PM_UPDATE=Y

NTAP_DFM_DATA_SET=itnapfs20:SC_SAP_SRM_Dev/DSR_Data,DSR_Log,DSR_Temp,DSR_Exe,DJR_Data,DJR_Log,DJR_Temp,DJR_Exe

NTAP_PM_RUN_BACKUP=Y

###################################

### Operations Manager Settings ###

###################################

OM_HOST=itromlx13.q8int.com

OM_USER=root

OM_PWD=pw

OM_PORT=8488

OM_EVENT_GENERATE=Y

This is the part of the log with the PM Update:

########## Running NetApp Protection Manager Backup Version Create for dataset SC_SAP_SRM_Dev ##########

[Fri Aug 12 04:32:06 2011] INFO: Registering snapshot BCK-daily_recent with Protection Manager dataset SC_SAP_SRM_Dev

[Fri Aug 12 04:32:06 2011] DEBUG: ZAPI REQUEST

<dp-backup-version-create>

        <backup-description>NetApp Snap Creator Framework Backup</backup-description>

        <dataset-name-or-id>SC_SAP_SRM_Dev</dataset-name-or-id>

        <retention-type>daily</retention-type>

        <version-timestamp>1313116263</version-timestamp>

        <version-members>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DSR_Data</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DSR_Log</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DSR_Temp</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DSR_Exe</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DJR_Data</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DJR_Log</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DJR_Temp</volume-name>

                </version-member-info>

                <version-member-info>

                        <snapshot-unique-id>1313116263</snapshot-unique-id>

                        <snapshot-name>BCK-daily_recent</snapshot-name>

                        <snapshot-contents></snapshot-contents>

                        <volume-name>itnapfs20:/DJR_Exe</volume-name>

                </version-member-info>

        </version-members>

</dp-backup-version-create>

[Fri Aug 12 04:32:06 2011] DEBUG: ZAPI RESULT

<results status="passed">

        <backup-id>3743</backup-id>

</results>

[Fri Aug 12 04:32:06 2011] INFO: Snapshot BCK-daily_recent for dataset SC_SAP_SRM_Dev registered with Protection Manager successfully

[ktenzer]

If SC registered the snapshot then the backup should be in protection manager. Click the dataset and "restore" button, then you can look if you have a primary backup with the SC snapshot naming.

Also please change SNAP_TIMESTAMP_ONLY=N to SNAP_TIMESTAMP_ONLY=Y

SC does not support _recent snapshots in protection manager. SC has to rename them and we dont rename them in PM just on filer so it creates problems. This could be causing your issue.

Regards,

Keith

[f_duranti]

From the logs I see that SC is registering the snapshot without error but it seems that protection manager will not go on with this registration.

I think there's something strange happening, it's ok that renaming should not work but at least I should see the recent snapshot.

Going on those dataset the "Restore" option is grayed out so I cannot restore them. I was expecting to see at least the last backup but it seems that SC report the backup is configured while PM don't register the backup.

In the afternoon I'll also try to configure a new backup to see what will happen. Strange also that PM will do the mirror, the job start and finish correctly (but is not reported inside the backup list too).

I've changed the parameter and will check this after 6 PM if the backup work.

Just another thing, on the sc 3.4.0 manual there's and example (A.9 on page 100)  that say to put NTAP_SNAPSHOT_NODELETE=Y when using protection manager.

Does this mean that backup/Snapshot will also be deleted by Protection Manager itself? In this case it will check for the protection policy to check how many hourly/daily/weekly backup to retain online?

There's any requirement with the snapshot naming policy of PM that can conflict with snapshot name of SC?

Thanks

Francesco

[ktenzer]

Ok then only thing I can think of is renaming the dataset is causing issues. I have never heard of anyone renaming dataset and it isnt something we test, if it works great but I cannot comment on that.

NTAP_SNAPSHOT_NODELETE=Y just means SC wont delete snapshots on primary. PM can per policy do this however PM seems to delete snapshots when it wants to not at any exact time so if you want that then having SC delete them is still best.

One thing to check is make sure PM policy daily or whatever which corresponds to SC policy isnt set to 0 or something, that could result in PM deleting snapshots as soon as they are created

Regards,

Keith

[f_duranti]

Well I'll do some check, eventually I'll create another not renamed SC dataset and I'll try to move the volume inside it and check if it works.

I've another dataset renamed that is working correctly from what I see but it was there from before the dfm 4 to oncommand 5 upgrade.

I also think that it "should work" or at least was thinked to work because i can specify in the SC configuration the Dataset names and also more then one dataset on the same backup configuration that could not be possible if it could only use the "pmsetup created" dataset.

I've checked and the PM policy don't have the daily (the policy I use) set to "0". The other policy are set to "0" (should i put those also on something higher or SC-PM exchange information on what policy they should use?). This also mean that I should have the PM policy always set higher or equal to the SC policy is this right?

Checking on snapshot they are available on the volume so PM did not delete them.

[ktenzer]

You do know onCommand 5.0 isn't supported in SC 3.4.0? We have done zero testing. It will be supported in next release (December) but I dont know if any code changes are needed. From what I hear there should be no code changes required, but I dont know.

I recommend you stay with DFM 4.x since that is what is supported and tested

If you do go with 5.0 then lets us know what works / doesn't, this will help in our testing as well

Regards,

Keith

[f_duranti]

SnapCreator 3.4.0 seems to work good with Oncommand.

The problem seems related on "case sensitive behaviour" but I don't know if it's a problem of OnCommand or of SnapCreator.

All my test that was working was on a storage that has the name in lowercase (itromfs10) and is saw from dfm 4.0/oncommand as lowercase, the one I have used now has the hostname in uppercase (ITNAPFS20) and is saw from oncommand as UPPERCASE.

In the configuration I've put the hostname in the Dataset configuration in lowercase:

NTAP_DFM_DATA_SET=itnapfs20:SC_SAP_SRM_Dev/DSR_Data,DSR_Log,DSR_Temp,DSR_Exe,DJR_Data,DJR_Log,DJR_Temp,DJR_Exe

after changing the hostname to UPPERCASE in the configuration I started to see the backup from protection manager (It's installed on a Linux Box, don't know if this can be part of the problem).

Now after changing the hostname on the config it works.

I think this can be a bug/bad behaviour, don't know if related to oncommand or snapcreator. What you think I can do? fill some kind of bug report?

Thanks

Francesco

[ktenzer]

Good catch!

This is an oncommand issue, it is handling host names as case sensitive. This isnt new though previous versions of oncommand were case sensitive w/host name.

SC could help out by auto-detecting hostname based on what is in dataset but that is something for future. Ill see if we can improve integration to handle this.

On another note I checked w/oncommand engineering and as far as SC is concerned nothing should have changed so we fully expect SC 3.4.0 to work w/oncomnand 5.0, we just havent finished testing.

Thanks for the help!

Regards,

Keith

[f_duranti]

Thanks to you Keith It's good to know that this can help ....

Just one last thing do you know if there's any way so that on oncommand the filer can be put in all uppercase or all lowercase letters and if this can be done without having to reconfigure all things in opmanager?

At this moment I've 4 of them lowercase and 2 of them uppercase and this will be really prone to create configuration problems in the future.

I think I should change the "hostname" of the storage but I don't know if this will be sufficient and what will happen (data loss on the oncommand or the need to delete and reconfigure every storage?)...

Any help will be really appreciated

Thanks

Francesco

[ktenzer · Accepted solution]

In theory oncommand should use name of filer so what you see when you logon to filer. When you add filer oncommand doesnt give option to specify name.

The only thing I can think of atm to change name is use oncommand cli. The "dfm host" command maybe do a "dfm host help", maybe there is option to change name? If that doesnt work maybe an NGS case, they might have a trick or two to help.

Let us know if you find solution

Have good weekened

Keith

[f_duranti]

I've finally renamed the host. I had to rename the host locally putting the right "hostname" with lowercase letter in /etc/rc and running "hostname lowercase" name on the storage.

After this I was able to do a dfm host rename (it was not working before because it was getting the uppercase from SNMP).

After this I had also to change the snapmirror.conf file with lowercase name on destination relationship or it was not updating snapmirror destination on the storage.

Thanks for the help

Francesco