No cluster license according to DFM?

bjornkoopmans · ‎2011-01-14

Hi,

I found a weird issue today with all of our ONTAP 8.0 controllers.

When I Check Cluster Configuration from the Cluster Console in Operations Manager on any of my 8.0 controllers, I get the following warnings:

---

Software Licenses
Error: <name> does not have a cluster license.
Error: <partner name> does not have a cluster license.
Action: Enable the cluster license on <name>.
Action: Enable the cluster license on <partner name>.

Error: The active/active controller may not fail over correctly.
---

However:

- proper cf license is entered

- cf status --> Cluster enabled, <partnername> is up.

- failover works fine

- my (only) ONTAP 7.3 active/active cluster doesn't display these warnings

- Cluster Console itself displays: Active/Active configuration enabled, <partner name> is up. Cluster enabled, <partner name> is up

- Diagnose Connectivity reports no issues

- I installed (and reinstalled) the ONTAP 8.0 DFM plugin

- the All Active/Active Controllers report does displays all controllers

That Operations Manager reports this issue isn't very problematic for us (we only found out about it today). However, we are just starting to use Provisioning manager and I'm unable to use provisioning policies with the Active/Active option checked since ProvMgr states that I don't have any Active/Active systems.

Does anyone have any suggestions?

Kind regards,

Bjorn

steinhorst · ‎2011-09-13

Did you figure this out? Having the exact same issue with DFM and ONTAP 8.0.2.

bjornkoopmans · ‎2011-09-13

Unfortunately, no. Even after upgrade to DOT 8.0.1P3. It seems you and me are the only ones experiencing this issue. :-S

sinhaa · ‎2011-09-14

Hello Bjorn,

The problem is not with your ONTAP version, but with Operations manager and hence provisioning manager ( they both use the same database). What is you DFM version that you are using? I believe this is one of the known issues and which has already been fixed in the latest version of DFM ( Now called OnCommand).

Provide the output of the commands:

1. dfm version

2. dfm about

Also its better to put question related to DFM or OnCommand in OnCommand Management Software portal to get quick responses. Here is the link

http://communities.netapp.com/community/products_and_solutions/storage_management_software?view=discussions&start=0

warm regards,

Abhishek

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

bjornkoopmans · ‎2011-09-14

Hi Abhishek,

I didn't realise I posted this in the ONTAP portal, so thanks for noticing!

>dfm version
dfbm.exe         4.0.2.7803 (4.0.2)
dfdrm.exe        4.0.2.7803 (4.0.2)
dfpm.exe         4.0.2.7803 (4.0.2)
dfm.exe          4.0.2.7803 (4.0.2)
dfmcheck.exe     4.0.2.7803 (4.0.2)
dfmconfig.exe    4.0.2.7803 (4.0.2)
dfmconsole.exe   4.0.2.7803 (4.0.2)
dfmmonitor.exe   4.0.2.7803 (4.0.2)
dfmperf.exe      4.0.2.7803 (4.0.2)
dfmscheduler.exe 4.0.2.7803 (4.0.2)
dfmserver.exe    4.0.2.7803 (4.0.2)
dfmwatchdog.exe 4.0.2.7803 (4.0.2)
eventd.exe       4.0.2.7803 (4.0.2)
grapher.exe      4.0.2.7803 (4.0.2)

>dfm about
Version 4.0.2.7803 (4.0.2)
Serial Number 1-50-017848

Administrator Name               domain\username
Host Name                        aa-sr-mg-1001
Host IP Address                  10.x.x.x
Host Full Name                   aa-sr-mg-1001.domain.extension
Operations Manager Node limit    11 (currently managing 12)
Provisioning Manager Node Limit 8 (currently managing 0)
Protection Manager Node Limit    8 (currently managing 0)
Operating System                 Microsoft Windows Server 2008 Enterprise Servic
e Pack 2 (Build 6002) x64 based
CPU Count                        8
System Memory                    16382 MB (load: 32%)
Installation Directory           C:/Program Files (x86)/IBM/DataFabric Manager/D
FM
                                 3.85 GB free (14.7%)
Perf Data Directory              M:\dfm\perfdata
Data Export Directory            D:\DFM\dataexport
Database Backup Directory        m:\dfm\db
Reports Archival Directory       D:\DFM\reports
Database Directory               m:/dfm/db
                                 18.1 GB free (45.5%)
Licensed Features                Operations Manager: installed
                                 Protection Manager: installed
                                 Provisioning Manager: installed

Installed Plugins                Storage System Config 6.5.1 (6.5.1) - storage systems and vFilers
                                 Storage System Config 6.5.2 (6.5.2) - storage systems and vFilers
                                 Storage System Config 6.5.3 (6.5.3) - storage systems and vFilers
                                 Storage System Config 6.5.4 (6.5.4) - storage systems and vFilers
                                 Storage System Config 6.5.5 (6.5.5) - storage systems and vFilers
                                 Storage System Config 6.5.6 (6.5.6) - storage systems and vFilers
                                 Storage System Config 6.5.7 (6.5.7) - storage systems and vFilers
                                 Storage System Config 7.0 (7.0.0.1) - storage systems and vFilers
                                 Storage System Config 7.0.1 (7.0.1.1) - storage systems and vFilers
                                 Storage System Config 7.0.2 (7.0.2) - storage systems and vFilers
                                 Storage System Config 7.0.3 (7.0.3) - storage systems and vFilers
                                 Storage System Config 7.0.4 (7.0.4) - storage systems and vFilers
                                 Storage System Config 7.0.5 (7.0.5) - storage systems and vFilers
                                 Storage System Config 7.0.6 (7.0.6) - storage systems and vFilers
                                 Storage System Config 7.0.7 (7.0.7) - storage systems and vFilers
                                 Storage System Config 7.1 (7.1.0.1) - storage systems and vFilers
                                 Storage System Config 7.1.1 (7.1.1.1) - storage systems and vFilers
                                 Storage System Config 7.1.2 (7.1.2.1) - storage systems and vFilers
                                 Storage System Config 7.1.3 (7.1.3) - storage systems and vFilers
                                 Storage System Config 7.2 (7.2) - storage systems and vFilers
                                 Storage System Config 7.2.1 (7.2.1.1) - storage systems and vFilers
                                 Storage System Config 7.2.2 (7.2.2) - storage systems and vFilers
                                 Storage System Config 7.2.3 (7.2.3) - storage systems and vFilers
                                 Storage System Config 7.2.4 (7.2.4) - storage systems and vFilers
                                 Storage System Config 7.2.5 (7.2.5.1) - storage systems and vFilers
                                 Storage System Config 7.2.6 (7.2.6.1) - storage systems and vFilers
                                 Storage System Config 7.2.7 (7.2.7) - storage systems and vFilers
                                 Storage System Config 7.3 (7.3) - storage systems and vFilers
                                 Storage System Config 7.3.1 (7.3.1.1) - storage systems and vFilers
                                 Storage System Config 7.3.2 (7.3.2) - storage systems and vFilers
                                 Storage System Config 7.3.3 (7.3.3RC1) - storage systems and vFilers
                                 Storage System Config 7.3.4 (7.3.4) - storage systems and vFilers
                                 Storage System Config 8.0 (8.0) - storage systems and vFilers
                                 Storage System Config 8.0.1 (8.0.1P3) - storage systems and vFilers

Kind regards,

Bjorn

sinhaa · ‎2011-09-14

Bjorn,

Your DFM version 4.0.2 should have the fix for this issue of not recognizing the cluster license for ONTAP 8.0.x.

Okay so lets start narrowing down the problem.

1. Are dfm monitors on your controllers working? Try 'dfm host diag <filer name or ip>' and see that there are no errors in the monitors.

2. List the output of 'dfm detail <filer name_id or ip>'

3. What is the exact dry-run error you are getting from provisioning manager? Can you give the putput or the screenshot.

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

bjornkoopmans · ‎2011-09-15

1. Here is the result of one of the systems (though a bit depersonified):

>dfm host diag toastera

Network Connectivity
IP Address             10.x.x.x
Network                (discovery disabled)
DNS Aliases            toastera.domain.extension
DNS Addresses          10.x.x.x,10.x.x.x
SNMP Version in Use    SNMPv1
SNMPv1                 Passed (135 ms)
SNMP Community         snmp_xxxx
SNMP sysName           toastera.domain.extension

SNMP sysObjectID       .1.3.6.1.4.1.789.2.3 (Clustered Filer)
SNMP productId         012345678
SNMPv3                 Passed (1 ms)
SNMPv3 Auth Protocol   MD5
SNMPv3 Privacy Enabled No
SNMPv3 Username        username
SNMP sysName           toastera.domain.extension
SNMP sysObjectID       .1.3.6.1.4.1.789.2.3 (Clustered Filer)
SNMP productId         012345678
ICMP Echo              Passed (4 ms)
HTTP                   Passed (1 ms)
NDMP Ping              Passed (port 10000, 0 ms)
NDMP Connect           Passed (1411 ms)
NDMP MD5 Passwd Check Passed
RSH                    Skipped (rshBinary is empty in global option)
SSH                    Passed (560 ms)
RLM                    Passed (8972 ms)
XML (https port 443)   Passed (15 ms)

Host Details
According to:   DataFabric Manager server       Host
Host Name       toastera.domain.extension      toastera.domain.extension
System ID       012345678                     012345678
Model           N6040                          N6040
Type            Clustered Storage System       Clustered Storage System
OS Version      8.0.1P3 7-Mode                 8.0.1P3 7-Mode
Revisions       330,8.0,2.1                    330,8.0,2.1

Monitoring Timestamps
Timestamp Name       Status   Interval     Default      Last Updated     Status
Error if older than ...
ccTimestamp          Normal   4 hours      4 hours
15 Sep 12:57
cfTimestamp          Normal   5 minutes    5 minutes    15 Sep 16:55     Normal
15 Sep 16:52
clusterTimestamp     Normal   15 minutes   15 minutes
15 Sep 16:42
cpuTimestamp         Normal   5 minutes    5 minutes    15 Sep 16:57     Normal
15 Sep 16:52
dfTimestamp          Normal   30 minutes   30 minutes   15 Sep 16:29     Normal
15 Sep 16:27
diskTimestamp        Normal   4 hours      4 hours      15 Sep 15:09     Normal
15 Sep 12:57
envTimestamp         Normal   5 minutes    5 minutes    15 Sep 16:53     Normal
15 Sep 16:52
fcTimestamp          Normal   5 minutes    5 minutes    15 Sep 16:51     Warning
15 Sep 16:52
fsTimestamp          Normal   15 minutes   15 minutes   15 Sep 16:47     Normal
15 Sep 16:42
hostPingTimestamp    Normal   1 minute     1 minute     15 Sep 16:56     Normal
15 Sep 16:56
ifTimestamp          Normal   15 minutes   15 minutes   15 Sep 16:47     Normal
15 Sep 16:42
licenseTimestamp     Error    1 day        4 hours      15 Sep 12:10     Normal
14 Sep 16:57
lunTimestamp         Normal   30 minutes   30 minutes   15 Sep 16:29     Normal
15 Sep 16:27
opsTimestamp         Normal   10 minutes   10 minutes   15 Sep 16:52     Normal
15 Sep 16:47
qtreeTimestamp       Normal   8 hours      8 hours
15 Sep 08:57
rbacTimestamp        Normal   1 day        1 day        15 Sep 12:19     Normal
14 Sep 16:57
userQuotaTimestamp   Normal   1 day        1 day        15 Sep 16:56     Normal
14 Sep 16:57
sanhostTimestamp     Normal   5 minutes    5 minutes
15 Sep 16:52
snapmirrorTimestamp Normal   30 minutes   30 minutes   15 Sep 16:30     Normal
15 Sep 16:27
snapshotTimestamp    Normal   30 minutes   30 minutes   15 Sep 16:38     Normal
15 Sep 16:27
statusTimestamp      Normal   10 minutes   10 minutes   15 Sep 16:48     Normal
15 Sep 16:47
sysInfoTimestamp     Normal   1 hour       1 hour       15 Sep 16:31     Normal
15 Sep 15:57
svTimestamp          Normal   30 minutes   30 minutes   15 Sep 16:41     Normal
15 Sep 16:27
svMonTimestamp       Normal   8 hours      8 hours
15 Sep 08:57
xmlQtreeTimestamp    Normal   8 hours      8 hours      15 Sep 16:27     Normal
15 Sep 08:57
vFilerTimestamp      Error    2 days       1 hour       15 Sep 12:06     Normal
13 Sep 16:57
vserverTimestamp     Normal   1 hour       1 hour
15 Sep 15:57

Performance Advisor Checklist
perfAdvisorEnabled     Passed
hostType               Passed
hostRevision           Passed
hostLogin              Passed
perfAdvisorTransport   Passed

2. Here the result of dfm detail of the same host:

>dfm detail toastera

agentTimestamp

asisIsLicensed Yes

autoClientStatEnabled

autoSupportIsEnabled 1

cacheConfigVersion

capacityKBytesTotal 35466986496

capacityKBytesUsed 33701888000

ccTimestamp

cf Yes

cf_remote No

cfCannotTakeoverCause 1

cfInterconnectStatus 4

cfPartnerLastStatusUpdate 1316099172

cfPartnerName toasterb

cfPartnerStatus 2

cfPartnerSysid 12345678

cfSettings 2

cfState 2

cfStatus

cfTimestamp 2011-09-15 17:06:14.000000

cifsIsLicensed Yes

cluster No

cluster_remote No

clusterId 81

clusterTimestamp

connTimestamp 2011-01-13 20:31:52.000000

cpuBusyThresholdInterval

cpuTimestamp 2011-09-15 17:02:37.000000

cpuTooBusyThreshold

dafsIsLicensed No

defaultvFilerInterface

dfTimestamp 2011-09-15 17:01:08.000000

diskActiveCount 121

diskFailedCount 0

diskFailedMessage There are no failed disks.

diskReconstructingCount 0

diskReconstructingParityCount 0

diskScrubbingCount 0

diskSpareCount 5

diskTimestamp 2011-09-15 15:09:00.000000

diskTotalCount 126

diskVerifyingParityCount 0

domainId 2

envTimestamp 2011-09-15 17:05:19.000000

fcpIsLicensed Yes

fcpOps 1932476429

fcpOpsPerSecond 660.881

fcpReadBytes 34115661179480

fcpReadBytesPerSecond 1.07658e+007

fcpWriteBytes 74132914166638

fcpWriteBytesPerSecond 3.12004e+006

fcTimestamp 2011-09-15 17:05:10.000000

fsTimestamp 2011-09-15 17:03:04.000000

hbaportTooBusyThreshold

hostAdminPort

hostAdminTransport https

hostAgentCIFSAccount

hostAgentCIFSPassword *****

hostAgentCIFSState Unknown

hostAgentFeatures 0

hostAgentOSSVEnabled No

hostAgentStatus Unknown

hostAgentVersion

hostAgentVersionNumber

hostCifsAllowWriteOplocksStatus Yes

hostCifsOplocksStatus Yes

hostCifsStatus Yes

hostCliCommand

hostCommunicationStatus Up

hostConfigValid No

hostConsoleAddress

hostDetailsRetrieved 1

hostEnableConfigMon No

hostEnableSNMPBasedQtreeMonitoring Yes

hostEnableUserQuotas Yes

hostEnvStatus Unknown

hostFcpStatus Yes

hostFullName toastera.domain.extension

hostFullOSVersionString Data ONTAP Release 8.0.1P3 7-Mode: Tue Mar 15

21:23:22 PDT 2011

hostHasSNMP Yes

hostId 81

hostIPbasedTPGroupsEnabled

hostIscsiStatus Yes

hostLanguages C:ar:cs:da:de:en:en_US:es:fi:fr:he:hr:hu:it:j

a:ja_v1:ja_JP.PCK:ja_JP.932:ja_JP.PCK_v2:ko:no:nl:pl:pt:ro:ru:sk:sl:sv:tr:zh:zh.

GBK:zh_TW:zh_TW.BIG5:

hostLogin root

hostLoginProtocol ssh

hostLoginState Okay

hostMaxBackupThreads

hostMustMonitorEnclosures No

hostNdmpLogin root

hostNdmpPassword *****

hostNdmpPort

hostNfsStatus Yes

hostNfsv3enabled 1

hostNfsv4enabled 0

hostOSVersion 8.0.1P3 7-Mode

hostPartnerId 84

hostPassword *****

hostPingMethod

hostPingStatus Up

hostPingTimestamp 2011-09-15 17:06:13.000000

hostPreferredAddr1

hostPreferredAddr2

hostPrimaryAddress 10.2.7.4

hostRevision 330

hostRLMAddress 10.2.8.100

hostRLMPingStatus Online

hostTimezone Europe/Amsterdam

hostType Clustered Filer

hostUuid

hostVfilerCurrentLimit 11

hostVfilerPlatformLimit 65

ifTimestamp 2011-09-15 17:03:23.000000

iscsiIsLicensed Yes

iscsiOps 12800

iscsiOpsPerSecond 0.00601504

iscsiReadBytes 0

iscsiReadBytesPerSecond 0

iscsiWriteBytes 800000

iscsiWriteBytesPerSecond 0.37594

licenseTimestamp 2011-09-15 12:10:43.000000

logfetchTimestamp 2010-05-18 08:25:15.000000

lunTimestamp 2011-09-15 17:01:15.000000

maxActiveDataTransfers

miscCacheAge 0

miscCifsOps 56258

miscCifsOpsPerSecond 0

miscGlobalStatus 3

miscGlobalStatusMessage The system's global status is normal.

miscHttpOps 96

miscHttpOpsPerSecond 0

miscNfsOps 0

miscNfsOpsPerSecond 0

miscTotalOps 1932545583

miscTotalOpsPerSecond 660.887

ndmpAccess all

ndmpCommunicationStatus up

ndmpCommunicationTimestamp 2011-09-15 16:57:30.000000

ndmpCredentialStatus Good

ndmpCredentialTimestamp 2011-09-15 16:57:30.000000

ndmpIsEnabled Yes

ndmpPingStatus Up

ndmpPingTimestamp 2011-09-15 16:57:30.000000

nearstoreIsLicensed Yes

nfsCacheIsLicensed No

nfsIsLicensed Yes

nisDomainId 0

nvramStatus 0

objComment

objCreationTimestamp 2010-04-01 08:35:37.000000

objDeleted

objDeletedBy

objDisappearedSince

objFlags 1245189

objFullName toastera.domain.extension

objId 81

objName toastera

objPerfStatus Normal

objPerfStatusCode 3

objStaleSince

objStatus Normal

objStatusChangeTimestamp 2011-09-15 05:07:21.000000

objStatusCode 3

objStatusMessage

objStatusTimestamp 2011-09-15 05:07:21.000000

objType Host

opsTimestamp 2011-09-15 17:03:19.000000

otherHostId

ownerName Companyname

perfAdvisorTransport httpsOk

perfDataExportEnabled

prefsnmpVersion 1

processHostIP

productApiUrl https://10.x.x.x:443/servlets/netapp.servlet

s.admin.XMLrequest_filer

productFirmwareVersion 4.1.0

productGuiUrl https://10.x.x.x:443/na_admin

productId 012345678

productModel N6040

productSerialNum 123456789012

qtreeTimestamp

rbacTimestamp 2011-09-15 12:19:49.000000

replicationMaximumTransfers 160

resetGrowthSamplesTimeStamp

rootStorageSystemId 93

sanhostClusterId

sanhostDriveMask 4294967295

sanhostIsClustered No

sanhostIsMPIO No

sanhostSnapDriveVersion

sanhostTimestamp

shareTimestamp 2011-09-15 16:51:04.000000

smIsLicensed Yes

smSyncIsLicensed Yes

snapmirrorAccess legacy

snapmirrorIsEnabled No

snapmirrorTimestamp 2011-09-15 17:01:21.000000

snapshotTimestamp 2011-09-15 16:38:52.000000

snapvaultAccess none

snapvaultIsEnabled Yes

srmhostTimestamp

statusTimestamp 2011-09-15 17:00:35.000000

svDrPrimaryIsLicensed No

svMonTimestamp

svPrimaryIsLicensed Yes

svSecondaryIsLicensed Yes

svSecondaryLinuxIsLicensed No

svSecondaryUnixIsLicensed No

svSecondaryWindowsIsLicensed No

svTimestamp 2011-09-15 16:41:51.000000

svWindowsOfmIsLicensed No

syncmirror_local No

sysContact Contactname

sysInfoTimestamp 2011-09-15 16:31:49.000000

sysLocation RC1

sysObjectID .1.3.6.1.4.1.789.2.3

sysUpTime 192028276

useHostsEquiv No

userQuotaTimestamp 2011-09-15 17:00:59.000000

vfHostId 81

vfId 81

vfilerIsLicensed Yes

vFilerTimestamp 2011-09-15 12:06:01.000000

vfIPSpace default-ipspace

vfName vfiler0

vfState Unknown

vserverTimestamp

xmlQtreeTimestamp 2011-09-15 16:27:59.000000

3. Weird thing is that I can't reproduce the error in provisioning manager anymore. It seems to work now. Allthough the DFM cluster console still states that both nodes don't have a cluster license!

adaikkap · ‎2011-09-15

The provisioning manger issue is fixed in 4.0.1 if I remeber correctly, where as the DFM issue is still there, pls raise a case and add it to the burt # 526035

Regards

adai

JNODTI21962 · ‎2012-04-12

Hi,

I have the same problem with OnCommand and DFM 5.0.

Weird?

Réal

adaikkap · ‎2012-04-12

Hi Real,

This bug526035 is fixed in 4.0.2D7 or in the upcoming 5.0.1 which is around the corner.

Regards

adai

REALWAITE · ‎2012-04-13

Cool,

Thanks Adai

Réal

jerome_barrelet · ‎2012-05-21

Hello Adai,

We are facing this problem with the newly integration of FAS 2040 with Ontap 8.0.3

We have upgraded to DFM 4.0.2 D7 suposed to fix the bug but we have still the problem.

The conformance is not working (see picture bellow)

if we have a look in all our filer in Active/Active page we have the problem in all filer in ontap version 8.x :

Do we have to do something more ? plugin are up to date.

Thanks in advance

Jerome

christin · ‎2012-05-22

Hi Jerome,

I found a few internal discussions that referenced BUG 411787. I recommend opening a support case to have your issue investigated.

Thanks,

Christine

sarvind · ‎2012-06-13

Hi

I tried Provisioning using Active-Active storage BOX and its successful OC 5.X

Can we have a snpmwalk of ths storage BOX
snmpwalk -v 1 -c public "Storage BOX IP" .1.3.6.1.4.1.789.1.2.3.8.0

If snmpwalk give a Integer value 3 which denotes InterconnectStatus is not proper w.r.t DFM. So check your physical connectivy of Storage BOX
And if the value is 4, the storage BOX are in correct state

Can we have an outpout of snmp walk for further analysis

Regards
Arvind

jerome_barrelet · ‎2012-06-19

Hello,

We have done the upgrade from DFM 4.0.2D7 to OnCommand 5.01 and the problem is still there.

A case is open we will see the answer of the support.

I'll post when I have something

Jerome

jerome_barrelet · ‎2012-06-25

Hello,

our problem have been solved by a Netapp case. here are the command to set on the DFM:

Problem Resolution:

set dfm options to default for cluster monitoring:

dfm option set ccMonInterval=

dfm option set cfMonInterval=

dfm option set clusterMonInterval=

dfm option set hostRBACMonInterval=

dfm option set backupDirMonInterval=

dfm option set vserverMonInterval=

Regards

Jerome

adaikkap · ‎2012-06-25

Hi Jerome,

Now I remember your environment. We customized it for Protection Manger Only Operations and during that time we disabled cfMonIntervel. From the above list except for dfm option set cfMonInterval= all others can be disabled.

As they have nothing to do with cluster status.

Regards

adai