Unable to set up Snapshot backups: Data not present on LUN

kofchur · ‎2013-06-24

HI all, I'm getting a bizzar "Data not present on LUN" message from dfm backup diag that seems to be hindering my setting up of snapshot backups:

Running:

OnCommand 5.1

Red Hat Enterprise Linux Server release 6.2

HUK 6.1

Snapdrive 5.2

DM-Multipath

ISCSI

[root@fgprd-oncommand-cdot-app001 dfm_data]# dfm backup create -t snapshot dfm_backup_test.sndb

Error: You can create a Snapshot based database backup only when the following

condition is met:

DataFabric Manager server data should reside on a dedicated SnapDrive managed LUN.

[root@fgprd-oncommand-cdot-app001 dfm_data]# dfm backup diag

Sybase Data Directory: /dfm_data/data/

Sybase Log Directory : /dfm_data/data/

Performance Data Directory: /dfm_data/perfdata

Script-plugins Directory: /dfm_data/script-plugins

Reports Archive Directory: /dfm_data/reports/

Plugins Directory: /dfm_data/plugins

Database Hosting System: localhost

SnapDrive Installed: Yes

SnapDrive Version: 5.2

SnapDrive Status: Passed(Data not present on LUN)

Volume Snapshot Status: NA

Backup Destination: /dfm_data/data/

Number of backups possible: 131

Backup Retention Count: no limit

Allowed Backup Type: Archive (ndb)

[root@fgprd-oncommand-cdot-app001 dfm_data]# pwd

/dfm_data

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls

conf data lost+found perfdata plugins reports script-plugins

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls /dfm_data/data

dfm_backup_2013-06-21_21-13-24.ndb monitordb.db monitordb.log

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls /dfm_data/perfdata

perf_20_138_12 perf_21_139_5 perf_22_140_2 perf_23_141_4 perf_24_353_19 perf_25_354_9 perf_26_355_14 perf_27_356_41 perf_29_138_4 perf_30_139_2 perf_32_130_5 perf_36_139_32

perf_20_139_12 perf_21_140_5 perf_22_141_2 perf_23_353_5 perf_24_354_19 perf_25_355_9 perf_26_356_14 perf_28_138_3 perf_29_139_4 perf_30_140_2 perf_32_136_5 perf_36_140_32

perf_20_140_12 perf_21_141_5 perf_22_353_3 perf_23_354_5 perf_24_355_19 perf_25_356_8 perf_27_138_43 perf_28_139_3 perf_29_140_4 perf_30_141_2 perf_33_130_3 perf_36_141_32

perf_20_141_12 perf_21_353_5 perf_22_354_3 perf_23_355_5 perf_24_356_18 perf_26_138_14 perf_27_139_41 perf_28_140_3 perf_29_141_4 perf_30_353_2 perf_33_136_4 perf_36_353_32

perf_20_353_15 perf_21_354_5 perf_22_355_3 perf_23_356_5 perf_25_138_8 perf_26_139_14 perf_27_140_42 perf_28_141_3 perf_29_353_4 perf_30_354_2 perf_34_130_7 perf_36_354_32

perf_20_354_15 perf_21_355_5 perf_22_356_3 perf_24_138_17 perf_25_139_8 perf_26_140_14 perf_27_141_43 perf_28_353_4 perf_29_354_4 perf_30_355_2 perf_34_136_7 perf_36_355_32

perf_20_355_15 perf_21_356_5 perf_23_138_4 perf_24_139_17 perf_25_140_8 perf_26_141_14 perf_27_353_43 perf_28_354_4 perf_29_355_4 perf_30_356_2 perf_35_130_14 perf_36_356_32

perf_20_356_15 perf_22_138_2 perf_23_139_4 perf_24_140_17 perf_25_141_8 perf_26_353_14 perf_27_354_43 perf_28_355_4 perf_29_356_4 perf_31_130_3 perf_35_136_15

perf_21_138_5 perf_22_139_2 perf_23_140_4 perf_24_141_17 perf_25_353_9 perf_26_354_14 perf_27_355_43 perf_28_356_4 perf_30_138_2 perf_31_136_3 perf_36_138_32

[root@fgprd-oncommand-cdot-app001 dfm_data]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 9.6G 5.1G 4.1G 56% /

tmpfs 5.9G 0 5.9G 0% /dev/shm

/dev/sda1 248M 30M 207M 13% /boot

10.2.9.200:/vol/asp/asp

200G 171G 29G 86% /mnt/asp

/dev/mapper/360a98000572d434964347274654d6154

50G 721M 47G 2% /dfm_data

[root@fgprd-oncommand-cdot-app001 dfm_data]# snapdrive storage show -all

WARNING: This operation can take several minutes
based on the configuration.

Connected LUNs and devices:

device filename        adapter path    size    proto   state   clone      lun path                     backing snapshot
----------------        ------- ----    ----    -----   -----   -----      --------                     ----------------
/dev/mapper/360a98000572d434964347274654d6154     -     P       50g     iscsi   online No         cse01b-va3:/vol/prd_iscsi_dfm/dfmdb_lun01    -

But as you can see, there is no missing data and all the appropriate directories are on the same iSCSI LUN and snapdrive can see it. Thanks in advanced!

-todd

kryan · ‎2013-06-24

Todd,

Where are the UM binaries installed? Can you share the output of "dfm about |grep Installation"?

How to configure Operations Manager snapshot backups.

https://kb.netapp.com/support/index?page=content&id=1011875

Thanks,

Kevin

kofchur · ‎2013-06-25

Yup. I made sure it was segregated:

[root@fgprd-oncommand-cdot-app001 ~]#

[root@fgprd-oncommand-cdot-app001 ~]# dfm about | grep -i installation

Installation Directory /opt/NTAPdfm

The only thing I can think of at the moment is that if there needs to be strict segregation of directories, maybe it is seeing someting (conf directory?) in my /dfm_data mount that should not be there:

[root@fgprd-oncommand-cdot-app001 dfm_data]# ls /dfm_data

conf data lost+found perfdata plugins reports script-plugins

But, this directory structure was a result of "dfm datastore setup", so I'm sceptical if this is the case.

adaikkap · ‎2013-06-25

Hi Todd,

Conf is definitely needs as it contains the encryption keys and its part of the db backup. Though I am still wondering what is missing.

Regards

adai

kofchur · ‎2013-06-25

So I thought maybe the problem was that I did not create the LUN with snapdrive, so I created another one with snapdrive and migrated the data over with dfm datastore setup and I'm still getting the same thing:

[root@fgprd-oncommand-cdot-app001 dfm_data]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 9.6G 5.2G 4.0G 57% /

tmpfs 5.9G 0 5.9G 0% /dev/shm

/dev/sda1 248M 30M 207M 13% /boot

10.2.9.200:/vol/asp/asp

200G 171G 29G 86% /mnt/asp

/dev/mapper/360a98000572d434964347274654d6154

50G 1022M 46G 3% /dfm_data <--------------------------Old DFM LUN not created by snapdrive

/dev/mapper/360a98000572d43496434727a70544372

50G 1021M 46G 3% /DFM_DATA <---------------------------New DFM LUN created by snapdrive

[root@fgprd-oncommand-cdot-app001 dfm_data]# dfm backup diag

Sybase Data Directory: /DFM_DATA/data/

Sybase Log Directory : /DFM_DATA/data/

Performance Data Directory: /DFM_DATA/perfdata

Script-plugins Directory: /DFM_DATA/script-plugins

Reports Archive Directory: /DFM_DATA/reports/

Plugins Directory: /DFM_DATA/plugins

Database Hosting System: localhost

SnapDrive Installed: Yes

SnapDrive Version: 5.2

SnapDrive Status: Passed(Data not present on LUN)

Volume Snapshot Status: NA

Backup Destination: /DFM_DATA/data/

Number of backups possible: 103

Backup Retention Count: no limit

Allowed Backup Type: Archive (ndb)

kryan · ‎2013-06-25

Todd,

I see evidence under bug 434447 that the combination of MPIO and iSCSI might be causing this problem.

Can you try this without MPIO or use an FCP LUN?

Thanks,

Kevin

kofchur · ‎2013-06-25

I was afraid that you might say something like that. I'll try it without MPIO. In the mean time I opened a case and sent dfmdc and autosupport data to NGS:

# 2004382947

adaikkap · ‎2013-06-26

Hi Todd,

This is more a SDU compatibility issue with MPIO with iSCSI in linux. There are 2 potential solution

Disable MPIO
Move to FCP

Decide which is easier

Regards

adai

kofchur · ‎2013-06-26

I wish it was that easy.

We only have NAS and iSCSI, so FCP is out. Disabling MPIO in this environment would not be good because we will be using WFA as a part of our automation process so we need the extra stability. I'm looking at the support matrix now and, according to this, there should not be any compatibility issues.

-todd

kryan · ‎2013-06-26

I believe that SDU supports MPIO and iSCSI, however the SDU interactions with UM with MPIO/iSCSI configuration is currently being treated as an RFE.

I have already added your support case to RFE 434447.

If you can't use single path or FCP, then you will need to use archive DFM backups.

kofchur · ‎2013-06-26

Fair enough. I'll just sit tight and do archive for now. Thanks for looking into it all. Most appreciated.

-todd

adaikkap · ‎2013-06-26

Hi Todd,

I think disabling MPIO for snapshot is much better than archive esp to reduce the backup time.

Regards

adai