Microsoft Virtualization Discussions

Operations Manager Efficiency Dashboard

J_ABELLA
5,346 Views

I have just upgraded to the new Datafabric manager, Version 5.1.0.15008 we have been running the

Operations Manager Storage Efficiency Dashboard Plugin hourly without issues for a few months. After the upgrade the script fails with no errors given.

I have tried uninstalling it and reinstalling it with no luck.

Our DFM server runs on Windows 2008.

Any advice on how to resolve it?

>dfm script list
ID   Script Name                                        Version
---- -------------------------------------------------- ----------
10245 Operations Manager Efficiency Dashboard            1.3.0

>dfm script job start 10245
Script Job 10253 started.

>dfm script job list
ID   Schedule Name             Script Name               Start Time   Status
---- ------------------------- ------------------------- ------------ ----------
10253 Manual                    Operations Manager Efficiency Dashboard 28 Aug 14:03 Errors
10251 OM - Dashboard (10246)    Operations Manager Efficiency Dashboard 28 Aug 13:56 Errors
10250 Manual                    Operations Manager Efficiency Dashboard 28 Aug 13:40 Errors
10248 OM - Dashboard (10246)    Operations Manager Efficiency Dashboard 28 Aug 13:33 Errors
10247 OM - Dashboard (10246)    Operations Manager Efficiency Dashboard 28 Aug 13:32 Errors

I do have these errros in the dfm error.log but the timestamps do not coincide with the running of the script:  The client IP is my current workstation IP.

[Tue Aug 28 09:47:15 2012] [notice] Child 3784: Listening on port 8080.

[Tue Aug 28 09:48:54 2012] [error] [client 10.230.31.48] File does not exist: D:/Program Files (x86)/DFM/web/Contents, referer: http://wn1151.ourdomain:8080/dashboard.html

[Tue Aug 28 13:24:57 2012] [error] [client 10.230.31.48] File does not exist: D:/Program Files (x86)/DFM/web/Contents, referer: http://wn1151.ourdomain:8080/dashboard.html

[Tue Aug 28 14:18:43 2012] [error] [client 10.230.31.48] File does not exist: D:/Program Files (x86)/DFM/web/Contents, referer: http://wn1151.ourdomain:8080/dashboard.html

1 ACCEPTED SOLUTION

J_ABELLA
5,346 Views

Well I was left to answer my own question but I thought I would share how I was able to resolve my issue for some other poor admin.  I attempted to open a ticket with Netapp support and was bombarded with information gathering before I could get any usefull analysis or even a technical discussion on what was occuring.  I understand the need to collect some preliminary information but found this hurtle to get assistance annoying at best.  Also IMHO they should have asked me to go ahead and run the DMFDC tool available in the downloads to collect this info instead of piece meal it manually.

So I don't want to bore you with my rant so I have placed Netapp's information request email below this but now I am going to tell you how I resolved my issue:

Since I had uninstalled and reinstalled the dfmeff script I found the D:\Program Files (x86)\DFM\script-plugins\dfmeff directory only had a few files contained in it.  When I went to run dfmeff.exe it stated it couldn not run becuase it couldn't find dbtool10.dll.  I performed a search for the file on my system C: and 😧 drives and couldn't locate the .dll.  I then performed a websearch to get an idea of what the .dll even is.  The .dll is part of SQL anywhere, I then returned to do another search but this time because I was lazy I did a search for only dbtool and found a dbtool11.dll.  I took a copy of the file and renamed the copy to dbtool10.dll and left it in the same directory, I then reran dfmeff.exe which uncompressed all the .xml files and now the efficiency dashboard script runs as expected.

------------------------------------------

Here is the email of information requested by support regarding my issue:

Summary of Environment

Note: All the commands referred below are to be
run on the DFM server CLI.

Framing Questions:

  1. What is the specific issue being experienced? Provide a detailed
    description along with the associated errors. Be verbose and collect examples.
    Include screen shots where necessary.
    • What are the steps leading to the specific issue?  This
      question is directed at understanding the workflow that leads to the specific
      issue and not just the issue itself.
    • Did this configuration ever work? If yes, when did it stop
      working?
    • What is the frequency of the failure? (How often, is it
      intermittent?)
    • Does it work in another setup/ configuration? (For example, a
      lab reproduction).
    • What changed since it worked previously?
    • When did the issue occur? Be as specific as possible, with the
      date and time; this will help NetApp when examining the log files.
    • What other errors, if any, are observed?
    • Does the issue affect every storage controller, some storage
      controllers, or only one storage controller? Identify which ones, by name,
      serial number and IP address.
    • Does the issue affect all the OnCommand Host Package clients,
      some OnCommand Host Package clients, or only one OnCommand Host Package client?
      Identify the specific client(s) by name and function.
    • Has isolation of the issue been attempted, such as suspending
      other services or operations to simplify the environment? Be as detailed as
      possible to establish timelines for the activity.
    • Have you referenced the product documentation, Knowledge Base or
      BUG Tools (public BURTS) to verify if the issue encountered is a known behavior
      or has a documented workaround?
  2. What are the DataFabric Manager server version, host platform
    and Operating System in use? Information can be found by using ‘dfm diag’ or
    ‘dfm about’ for CLI methods or by going to
    Help > About, and
    then clicking
    Configuration for the UI method.
  3. Are you running OnCommand Standard or the Express edition?
    Are you running previous release of Operations Manager (if so, what is the
    release)?
  4. Which release of Data ONTAP is the affected storage controller
    running?
  5. Are you running a supported configuration as specified in the
    OnCommand Core and Host Package Installation and Setup Guide and/ or the NetApp
    Interoperability Matrix Tool?
  6. Does the OnCommand server meet the minimum requirements as
    specified within the OnCommand Core and Host Package Installation and Setup Guide
    and/or NetApp Interoperability Matrix Tool?
  7. Is the DataFabric Manager server deployed on a system that is
    running no other applications?

    Note:
    Best practices and product documentation indicate that the DataFabric Manager
    server be the only application running on the server.
  8. If OnCommand worked properly before the reported behavior, were
    any changes made to the storage system or the OnCommand server, such as patches
    to the host OS, product reconfiguration, network, or other maintenance?
  9. Is there a firewall installed on the network that could be
    affecting communication between the storage system and the DataFabric Manager
    server?
  10. Could this be a browser issue? Are you using one of the
    supported browsers? Have you tried clearing the browser cache?

    Note: OnCommand
    Core Package supports a reduced set of browser and versions as compared to the
    existing DFM product. Check the NetApp Interoperability Matrix Tool for a list of supported browsers and
    versions.

SECTION B

Data Collection:

Given the complex nature of
this product with regard to all the integrated pieces, collecting all logs in
at the same time for inspection after the issue has been observed is required
to continue troubleshooting.  It may be necessary to capture data in
multiple iterations, and it is best to capture data from all the involved
systems at the same time for inspection.

  1. If the DataFabric Manager server operating system is Windows,
    execute
    msinfo32.exe, save
    the output as a
    *.nfo file,
    and upload the archive. For more information, see article  1010090: How to upload a file to NetApp.
  2. Download and extract the DataFabric Manager Data Collector
    (dfmdc) from the Utility Toolchest in the Downloads link in the NetApp Support site,
    to the DataFabric Manager server, and run it per instructions on the download
    page. If the host is Windows, dfmdc must be run as Administrator from the CLI
    and not be impacted by UAC. Right-click the command prompt shortcut, and select
    Run as Administrator. Navigate to the
    directory where the dfmdc data files were extracted and begin the data
    collection. The dfmdc tool creates a single file archive as the output. Upload
    the archive per Section - 3: Sending Data to NetApp, below.
  3. Collect the Host Services information from the DataFabric
    Manager server. Archive/ ZIP the following into a single file named
    hostserviceinfo, and add the appropriate
    extension. Upload the archive per Section - 3: Sending Data to NetApp, below.
    • dfm
      hs list-a
    • dfm
      hs diag <hostserviceID> (column one from dfm hs list is the ID)
    • dfm
      hs controller list
    • dfm
      hs plugin list
  4. Collect the Virtual Sever Object information from the DataFabric
    Manager server. Archive/ ZIP the following into a single file name
    vservinof and add
    the appropriate extension. Upload the archive per Section - 3: Sending Data to
    NetApp, below.
    • dfm
      vm list –a
    • dfm
      vmdatastore list –a
    • dfm
      vcenter list –a
    • dfm
      hypervisor list –a
    • dfm
      datacenter list –a
    • dfm
      vdisk list -a
  5. Generate an AutoSupport from all storage controllers involved in
    the issue. Execute the command at the same time as the data collection process
    is in progress, and tag the AutoSupport with the case number (
    <> are delimiters and should not be entered in
    the actual command):


    options autosupport.doit <casenumber>
  6. Download and run the Data Collection Tool for Windows
    (ONTAPWinDC) from the Utility Toolchest in the Downloads link in the NetApp Support site,
    to each host involved in the issue (each host would be where the OnCommand Host
    Package was installed for integration with this DataFabric Manager server).
    Upload the archive per Section - 3: Sending Data to NetApp, below.
  7. Collect the following logs from the same hosts where ONTAPWinDC
    was executed. Archive/ ZIP the following into a single file named
    hostpkglogs, and add the appropriate
    extension. Upload the archive per Section - 3: Sending Data to NetApp, below.
    • C:\Program
      Files NetApp\OnCommand Host Package\VMware Plugin\log\*
    • C:\Program
      Files NetApp\OnCommand Host Package\VMware UI Plugin\log\*
    • C:\Program
      Files NetApp\OnCommand Host Package\SnapDrive\snpdrvdbg.log
  8. If the issue involves the NetApp Host Agent (NHA), collect the
    following:
    • The NHA, by default, is installed in:
      • C:\Program Files\Network
        Appliance\Agent\
        on
        Windows (older DFM releases)
      • C:\Program
        Files\NetApp\DataFabric\Agent\
        on
        Windows (newer DFM releases)
      • /opt/NTAPagent/ on Solaris and Linux
    • Collect the output of the following commands:
      • Windows: ntap_agent.exe query and ntap_agent.exe version
      • Linux/
        Solaris:
        ntap_agent query and ntap_agent version
    • Collect ntap_agent.log file (Windows).
    • Collect ntap_agent.logntap_agent.acl, and ntap_agent.cfg files (Linux/ Solaris).
    • Combine all files/ output into a folder named 'nha' and upload
      the archive per Section - 3: Sending Data to NetApp, below.
  9. If the issue involves the NetApp Management Console (NMC),
    collect the following:
    • The NMC, by default, is installed in:
      • C:\Program Files\Network
        Appliance\Management Console\
        on
        Windows (older DFM releases)
      • C:\Program
        Files\NetApp\DataFabric\Management Console\
        on Windows (newer DFM releases)
      • <dir_where_the_downloaded_installer_is_run> on Solaris and Linux
    • Collect the following files/ directories from the <installdir> on Windows and Linux/ Solaris:
      • Entire logs dir, boot.propertieslog4j.properties, and vendor.properties files
      • If
        present, all
        hs_err_*****.log files
    • Combine all files/ output into a folder named 'nmc' and upload
      the archive per Section - 3: Sending Data to NetApp, below.
  10. Depending on archive file sizes, the individual archives above
    might be combined into a single archive for upload. Do not mix individual
    files; rather combine the individual archives into a single master archive for
    a particular data collection request.

View solution in original post

4 REPLIES 4

J_ABELLA
5,347 Views

Well I was left to answer my own question but I thought I would share how I was able to resolve my issue for some other poor admin.  I attempted to open a ticket with Netapp support and was bombarded with information gathering before I could get any usefull analysis or even a technical discussion on what was occuring.  I understand the need to collect some preliminary information but found this hurtle to get assistance annoying at best.  Also IMHO they should have asked me to go ahead and run the DMFDC tool available in the downloads to collect this info instead of piece meal it manually.

So I don't want to bore you with my rant so I have placed Netapp's information request email below this but now I am going to tell you how I resolved my issue:

Since I had uninstalled and reinstalled the dfmeff script I found the D:\Program Files (x86)\DFM\script-plugins\dfmeff directory only had a few files contained in it.  When I went to run dfmeff.exe it stated it couldn not run becuase it couldn't find dbtool10.dll.  I performed a search for the file on my system C: and 😧 drives and couldn't locate the .dll.  I then performed a websearch to get an idea of what the .dll even is.  The .dll is part of SQL anywhere, I then returned to do another search but this time because I was lazy I did a search for only dbtool and found a dbtool11.dll.  I took a copy of the file and renamed the copy to dbtool10.dll and left it in the same directory, I then reran dfmeff.exe which uncompressed all the .xml files and now the efficiency dashboard script runs as expected.

------------------------------------------

Here is the email of information requested by support regarding my issue:

Summary of Environment

Note: All the commands referred below are to be
run on the DFM server CLI.

Framing Questions:

  1. What is the specific issue being experienced? Provide a detailed
    description along with the associated errors. Be verbose and collect examples.
    Include screen shots where necessary.
    • What are the steps leading to the specific issue?  This
      question is directed at understanding the workflow that leads to the specific
      issue and not just the issue itself.
    • Did this configuration ever work? If yes, when did it stop
      working?
    • What is the frequency of the failure? (How often, is it
      intermittent?)
    • Does it work in another setup/ configuration? (For example, a
      lab reproduction).
    • What changed since it worked previously?
    • When did the issue occur? Be as specific as possible, with the
      date and time; this will help NetApp when examining the log files.
    • What other errors, if any, are observed?
    • Does the issue affect every storage controller, some storage
      controllers, or only one storage controller? Identify which ones, by name,
      serial number and IP address.
    • Does the issue affect all the OnCommand Host Package clients,
      some OnCommand Host Package clients, or only one OnCommand Host Package client?
      Identify the specific client(s) by name and function.
    • Has isolation of the issue been attempted, such as suspending
      other services or operations to simplify the environment? Be as detailed as
      possible to establish timelines for the activity.
    • Have you referenced the product documentation, Knowledge Base or
      BUG Tools (public BURTS) to verify if the issue encountered is a known behavior
      or has a documented workaround?
  2. What are the DataFabric Manager server version, host platform
    and Operating System in use? Information can be found by using ‘dfm diag’ or
    ‘dfm about’ for CLI methods or by going to
    Help > About, and
    then clicking
    Configuration for the UI method.
  3. Are you running OnCommand Standard or the Express edition?
    Are you running previous release of Operations Manager (if so, what is the
    release)?
  4. Which release of Data ONTAP is the affected storage controller
    running?
  5. Are you running a supported configuration as specified in the
    OnCommand Core and Host Package Installation and Setup Guide and/ or the NetApp
    Interoperability Matrix Tool?
  6. Does the OnCommand server meet the minimum requirements as
    specified within the OnCommand Core and Host Package Installation and Setup Guide
    and/or NetApp Interoperability Matrix Tool?
  7. Is the DataFabric Manager server deployed on a system that is
    running no other applications?

    Note:
    Best practices and product documentation indicate that the DataFabric Manager
    server be the only application running on the server.
  8. If OnCommand worked properly before the reported behavior, were
    any changes made to the storage system or the OnCommand server, such as patches
    to the host OS, product reconfiguration, network, or other maintenance?
  9. Is there a firewall installed on the network that could be
    affecting communication between the storage system and the DataFabric Manager
    server?
  10. Could this be a browser issue? Are you using one of the
    supported browsers? Have you tried clearing the browser cache?

    Note: OnCommand
    Core Package supports a reduced set of browser and versions as compared to the
    existing DFM product. Check the NetApp Interoperability Matrix Tool for a list of supported browsers and
    versions.

SECTION B

Data Collection:

Given the complex nature of
this product with regard to all the integrated pieces, collecting all logs in
at the same time for inspection after the issue has been observed is required
to continue troubleshooting.  It may be necessary to capture data in
multiple iterations, and it is best to capture data from all the involved
systems at the same time for inspection.

  1. If the DataFabric Manager server operating system is Windows,
    execute
    msinfo32.exe, save
    the output as a
    *.nfo file,
    and upload the archive. For more information, see article  1010090: How to upload a file to NetApp.
  2. Download and extract the DataFabric Manager Data Collector
    (dfmdc) from the Utility Toolchest in the Downloads link in the NetApp Support site,
    to the DataFabric Manager server, and run it per instructions on the download
    page. If the host is Windows, dfmdc must be run as Administrator from the CLI
    and not be impacted by UAC. Right-click the command prompt shortcut, and select
    Run as Administrator. Navigate to the
    directory where the dfmdc data files were extracted and begin the data
    collection. The dfmdc tool creates a single file archive as the output. Upload
    the archive per Section - 3: Sending Data to NetApp, below.
  3. Collect the Host Services information from the DataFabric
    Manager server. Archive/ ZIP the following into a single file named
    hostserviceinfo, and add the appropriate
    extension. Upload the archive per Section - 3: Sending Data to NetApp, below.
    • dfm
      hs list-a
    • dfm
      hs diag <hostserviceID> (column one from dfm hs list is the ID)
    • dfm
      hs controller list
    • dfm
      hs plugin list
  4. Collect the Virtual Sever Object information from the DataFabric
    Manager server. Archive/ ZIP the following into a single file name
    vservinof and add
    the appropriate extension. Upload the archive per Section - 3: Sending Data to
    NetApp, below.
    • dfm
      vm list –a
    • dfm
      vmdatastore list –a
    • dfm
      vcenter list –a
    • dfm
      hypervisor list –a
    • dfm
      datacenter list –a
    • dfm
      vdisk list -a
  5. Generate an AutoSupport from all storage controllers involved in
    the issue. Execute the command at the same time as the data collection process
    is in progress, and tag the AutoSupport with the case number (
    <> are delimiters and should not be entered in
    the actual command):


    options autosupport.doit <casenumber>
  6. Download and run the Data Collection Tool for Windows
    (ONTAPWinDC) from the Utility Toolchest in the Downloads link in the NetApp Support site,
    to each host involved in the issue (each host would be where the OnCommand Host
    Package was installed for integration with this DataFabric Manager server).
    Upload the archive per Section - 3: Sending Data to NetApp, below.
  7. Collect the following logs from the same hosts where ONTAPWinDC
    was executed. Archive/ ZIP the following into a single file named
    hostpkglogs, and add the appropriate
    extension. Upload the archive per Section - 3: Sending Data to NetApp, below.
    • C:\Program
      Files NetApp\OnCommand Host Package\VMware Plugin\log\*
    • C:\Program
      Files NetApp\OnCommand Host Package\VMware UI Plugin\log\*
    • C:\Program
      Files NetApp\OnCommand Host Package\SnapDrive\snpdrvdbg.log
  8. If the issue involves the NetApp Host Agent (NHA), collect the
    following:
    • The NHA, by default, is installed in:
      • C:\Program Files\Network
        Appliance\Agent\
        on
        Windows (older DFM releases)
      • C:\Program
        Files\NetApp\DataFabric\Agent\
        on
        Windows (newer DFM releases)
      • /opt/NTAPagent/ on Solaris and Linux
    • Collect the output of the following commands:
      • Windows: ntap_agent.exe query and ntap_agent.exe version
      • Linux/
        Solaris:
        ntap_agent query and ntap_agent version
    • Collect ntap_agent.log file (Windows).
    • Collect ntap_agent.logntap_agent.acl, and ntap_agent.cfg files (Linux/ Solaris).
    • Combine all files/ output into a folder named 'nha' and upload
      the archive per Section - 3: Sending Data to NetApp, below.
  9. If the issue involves the NetApp Management Console (NMC),
    collect the following:
    • The NMC, by default, is installed in:
      • C:\Program Files\Network
        Appliance\Management Console\
        on
        Windows (older DFM releases)
      • C:\Program
        Files\NetApp\DataFabric\Management Console\
        on Windows (newer DFM releases)
      • <dir_where_the_downloaded_installer_is_run> on Solaris and Linux
    • Collect the following files/ directories from the <installdir> on Windows and Linux/ Solaris:
      • Entire logs dir, boot.propertieslog4j.properties, and vendor.properties files
      • If
        present, all
        hs_err_*****.log files
    • Combine all files/ output into a folder named 'nmc' and upload
      the archive per Section - 3: Sending Data to NetApp, below.
  10. Depending on archive file sizes, the individual archives above
    might be combined into a single archive for upload. Do not mix individual
    files; rather combine the individual archives into a single master archive for
    a particular data collection request.

allison
5,346 Views

Sorry you had a less than stellar experience with our Support team. I can imagine how frustrated you were and we have passed this information along to our Support team. Thank you for sharing your knowledge with the Community for others who may encounter the same issue. Please feel free to reach out to me if you have any other feedback or concerns. Thanks.

~allison

allison
5,346 Views

James Abella We would like to follow-up on this case. Can you give me your case support number for to be investigated further? Thanks.

J_ABELLA
5,346 Views

Sure Allison, here you go: 2003494058.

Public