Subscribe

Unable to set up Snapshot backups: Data not present on LUN

HI all,  I'm getting a bizzar "Data not present on LUN" message from dfm backup diag that seems to be hindering my setting up of snapshot backups:

Running:

OnCommand 5.1

Red Hat Enterprise Linux Server release 6.2

HUK 6.1

Snapdrive 5.2

DM-Multipath

ISCSI

[root@fgprd-oncommand-cdot-app001 dfm_data]# dfm backup create -t snapshot dfm_backup_test.sndb

Error: You can create a Snapshot based database backup only when the following

condition is met:

DataFabric Manager server data should reside on a dedicated SnapDrive managed LUN.

[root@fgprd-oncommand-cdot-app001 dfm_data]# dfm backup diag

Sybase Data Directory:      /dfm_data/data/

Sybase Log Directory :      /dfm_data/data/

Performance Data Directory: /dfm_data/perfdata

Script-plugins Directory:   /dfm_data/script-plugins

Reports Archive Directory:  /dfm_data/reports/

Plugins Directory:          /dfm_data/plugins

Database Hosting System:    localhost

SnapDrive Installed:        Yes

SnapDrive Version:          5.2

SnapDrive Status:           Passed(Data not present on LUN)

Volume Snapshot Status:     NA

Backup Destination:         /dfm_data/data/

Number of backups possible: 131

Backup Retention Count:     no limit

Allowed Backup Type:        Archive (ndb)

[root@fgprd-oncommand-cdot-app001 dfm_data]# pwd

/dfm_data

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls

conf  data  lost+found  perfdata  plugins  reports  script-plugins

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls /dfm_data/data

dfm_backup_2013-06-21_21-13-24.ndb  monitordb.db  monitordb.log

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls /dfm_data/perfdata

perf_20_138_12  perf_21_139_5  perf_22_140_2  perf_23_141_4   perf_24_353_19  perf_25_354_9   perf_26_355_14  perf_27_356_41  perf_29_138_4  perf_30_139_2  perf_32_130_5   perf_36_139_32

perf_20_139_12  perf_21_140_5  perf_22_141_2  perf_23_353_5   perf_24_354_19  perf_25_355_9   perf_26_356_14  perf_28_138_3   perf_29_139_4  perf_30_140_2  perf_32_136_5   perf_36_140_32

perf_20_140_12  perf_21_141_5  perf_22_353_3  perf_23_354_5   perf_24_355_19  perf_25_356_8   perf_27_138_43  perf_28_139_3   perf_29_140_4  perf_30_141_2  perf_33_130_3   perf_36_141_32

perf_20_141_12  perf_21_353_5  perf_22_354_3  perf_23_355_5   perf_24_356_18  perf_26_138_14  perf_27_139_41  perf_28_140_3   perf_29_141_4  perf_30_353_2  perf_33_136_4   perf_36_353_32

perf_20_353_15  perf_21_354_5  perf_22_355_3  perf_23_356_5   perf_25_138_8   perf_26_139_14  perf_27_140_42  perf_28_141_3   perf_29_353_4  perf_30_354_2  perf_34_130_7   perf_36_354_32

perf_20_354_15  perf_21_355_5  perf_22_356_3  perf_24_138_17  perf_25_139_8   perf_26_140_14  perf_27_141_43  perf_28_353_4   perf_29_354_4  perf_30_355_2  perf_34_136_7   perf_36_355_32

perf_20_355_15  perf_21_356_5  perf_23_138_4  perf_24_139_17  perf_25_140_8   perf_26_141_14  perf_27_353_43  perf_28_354_4   perf_29_355_4  perf_30_356_2  perf_35_130_14  perf_36_356_32

perf_20_356_15  perf_22_138_2  perf_23_139_4  perf_24_140_17  perf_25_141_8   perf_26_353_14  perf_27_354_43  perf_28_355_4   perf_29_356_4  perf_31_130_3  perf_35_136_15

perf_21_138_5   perf_22_139_2  perf_23_140_4  perf_24_141_17  perf_25_353_9   perf_26_354_14  perf_27_355_43  perf_28_356_4   perf_30_138_2  perf_31_136_3  perf_36_138_32

[root@fgprd-oncommand-cdot-app001 dfm_data]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             9.6G  5.1G  4.1G  56% /

tmpfs                 5.9G     0  5.9G   0% /dev/shm

/dev/sda1             248M   30M  207M  13% /boot

10.2.9.200:/vol/asp/asp

                      200G  171G   29G  86% /mnt/asp

/dev/mapper/360a98000572d434964347274654d6154

                       50G  721M   47G   2% /dfm_data

[root@fgprd-oncommand-cdot-app001 dfm_data]# snapdrive storage show -all

WARNING: This operation can take several minutes
          based on the configuration.

Connected LUNs and devices:

device filename        adapter path    size    proto   state   clone      lun path                     backing snapshot
----------------        ------- ----    ----    -----   -----   -----      --------                     ----------------
/dev/mapper/360a98000572d434964347274654d6154     -     P       50g     iscsi   online  No         cse01b-va3:/vol/prd_iscsi_dfm/dfmdb_lun01    -

But as you can see, there is no missing data and all the appropriate directories are on the same iSCSI LUN and snapdrive can see it.  Thanks in advanced!

-todd

Re: Unable to set up Snapshot backups: Data not present on LUN

Todd,

Where are the UM binaries installed?  Can you share the output of "dfm about |grep Installation"?

How to configure Operations Manager snapshot backups.

https://kb.netapp.com/support/index?page=content&id=1011875

Thanks,

Kevin

Re: Unable to set up Snapshot backups: Data not present on LUN

Yup.  I made sure it was segregated:

[root@fgprd-oncommand-cdot-app001 ~]#

[root@fgprd-oncommand-cdot-app001 ~]# dfm about | grep -i installation

Installation Directory           /opt/NTAPdfm

The only thing I can think of at the moment is that if there needs to be strict segregation of directories, maybe it is seeing someting (conf directory?) in my /dfm_data mount that should not be there:

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls /dfm_data

conf  data  lost+found  perfdata  plugins  reports  script-plugins

But, this directory structure was a result of "dfm datastore setup", so I'm sceptical if this is the case.

Re: Unable to set up Snapshot backups: Data not present on LUN

Hi Todd,

    Conf is definitely needs as it contains the encryption keys and its part of the db backup. Though I am still wondering what is missing.

Regards

adai

Re: Unable to set up Snapshot backups: Data not present on LUN

So I thought maybe the problem was that I did not create the LUN with snapdrive, so I created another one with snapdrive and migrated the data over with dfm datastore setup and I'm still getting the same thing:

[root@fgprd-oncommand-cdot-app001 dfm_data]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             9.6G  5.2G  4.0G  57% /

tmpfs                 5.9G     0  5.9G   0% /dev/shm

/dev/sda1             248M   30M  207M  13% /boot

10.2.9.200:/vol/asp/asp

                      200G  171G   29G  86% /mnt/asp

/dev/mapper/360a98000572d434964347274654d6154

                       50G 1022M   46G   3% /dfm_data               <--------------------------Old DFM LUN not created by snapdrive

/dev/mapper/360a98000572d43496434727a70544372

                       50G 1021M   46G   3% /DFM_DATA          <---------------------------New DFM LUN created by snapdrive

[root@fgprd-oncommand-cdot-app001 dfm_data]# dfm backup diag

Sybase Data Directory:      /DFM_DATA/data/

Sybase Log Directory :      /DFM_DATA/data/

Performance Data Directory: /DFM_DATA/perfdata

Script-plugins Directory:   /DFM_DATA/script-plugins

Reports Archive Directory:  /DFM_DATA/reports/

Plugins Directory:          /DFM_DATA/plugins

Database Hosting System:    localhost

SnapDrive Installed:        Yes

SnapDrive Version:          5.2

SnapDrive Status:           Passed(Data not present on LUN)

Volume Snapshot Status:     NA

Backup Destination:         /DFM_DATA/data/

Number of backups possible: 103

Backup Retention Count:     no limit

Allowed Backup Type:        Archive (ndb)

Re: Unable to set up Snapshot backups: Data not present on LUN

Todd,

I see evidence under bug 434447 that the combination of MPIO and iSCSI might be causing this problem.

Can you try this without MPIO or use an FCP LUN?

Thanks,

Kevin

Re: Unable to set up Snapshot backups: Data not present on LUN

I was afraid that you might say something like that.  I'll try it without MPIO.  In the mean time I opened a case and sent dfmdc and autosupport data to NGS:

# 2004382947

Re: Unable to set up Snapshot backups: Data not present on LUN

Hi Todd,

     This is more a SDU compatibility issue with MPIO with iSCSI in linux. There are 2 potential solution

  1. Disable MPIO
  2. Move to FCP

Decide which is easier

Regards

adai

Re: Unable to set up Snapshot backups: Data not present on LUN

I wish it was that easy.

We only have NAS and iSCSI, so FCP is out.  Disabling MPIO in this environment would not be good because we will be using WFA as a part of our automation process so we need the extra stability.  I'm looking at the support matrix now and, according to this, there should not be any compatibility issues.

-todd

Re: Unable to set up Snapshot backups: Data not present on LUN

I believe that SDU supports MPIO and iSCSI, however the SDU interactions with UM with MPIO/iSCSI configuration is currently being treated as an RFE.

I have already added your support case to RFE 434447.

If you can't use single path or FCP, then you will need to use archive DFM backups.