WFA DB information not updated after successful acquisition

[DavidSpano]

The data source form my OnCommand Unified Manager (5.2 7mode) seems to be setup correctly gives a "Completed" return message when it runs.  However when I changed a Qtree quota size limit on the filer and ran a acquisition, the workflow dropdown for the qtree still shows the old Qtree limit size.  Realizing that DFM has to poll the filer to get the updated information, I tested DFM connect to the filer which was good.  I also waited a few days as I was not sure of the polling interval for DFM for qtree information.  I am very new to WFA server and need some suggestions on how to trouble shoot this problem.

[korns]

Hey David,

Hummm. One thing I've noticed is inside the 7-Mode data scheme there are separate tables for qtree and quota. They both have fields for a disk_limit_mb. They are called:

In the qtree table there is a field called: disk_limit_mb

In the quota table there is a field called: hard_disk_limit_mb

I'm not entirely sure why both exist. For a couple of qtrees I'm looking at both fields have the exact same values. I might have thought that the quota table reflected what was in the /etc/quota file and I wouldn't have suspected to see anything like a  disk_limit_mb field on the qtree table itself ... since qtrees don't really hold 'quota limit' information, because instead it is really in the /etc/quota file. So my guess is the value in the qtree table is just a copy of that same thing that is in the quota table ... just included b WFA engineers since it is so common for folks to put disk limits on things and it saves writing tricky SQL if it is also just right there in the qtree table.

I suspect the workflow you are looking at (actually, I know 🙂 uses the disk_limit_mb from the qtree table. When you say you changed a quota size on the filer does that mean you edited the /etc/quota file?

Here are some things I would think of to try and look into:

- I wonder if turning quotas off/on on the volume is required for the change to show up

- don't forget to force a OCUM polling update (filer->OCUM), wait a bit, then force a WFA acquire (OCUM->WFA)

- If you can, peek at the data dictionary tables inside WFA to look at both qtree and quota tables

I'll see if I can re-create the problem also.

[korns]

The way I force OCUM 5.2 to re-poll is go to Storage tab, view controllers, highlight the filer/controller in question and hit the [Refresh Monitoring] button.

The way I force WFA 2.x to re-poll is go to Execution tab, highlight the Data Source for this OCUM, right-click and pick 'Acquire Now'

[DavidSpano]

Yes, I changed the quota on the test qtree in the /etc/quotas file; then I turn off quotas and turned them back on.

I looked in the WFA Cache DB using the HeidiSQL browser and the disk_limit in both the "qtree" and "quota" tables contained the old value of 5000MB.

While it has been days since I manually made the change, I will OCUM to re-poll the storage system and then perform a WFA acquire to see what happens.

Thanks.

[DavidSpano]

I did a "Refresh Monitoring" on the OCUM server for the controller in question.  I then ran the OCUM Qtree report and it listed the qtree with the correct disk_Limit.  I then ran the aquire on the WFA server, but the qtree disk_limit size did not update in the WFA Cache_dB.

[DavidSpano]

This problem seems like the "aquire" is failing even though the Job Status show complete with no error messages.  I looked at the wfa_acquisition.log, wfa.log, and server.log and they all give warnings when the "aquire" runs that says:

2014-05-16 16:28:43,053 INFO [com.netapp.wfa.cache.job.CacheJobExecutorImpl] (Thread-9484 (HornetQ-client-global-threads-1319482573)) Acquisition job 15439 - 6 warnings - Duplicate rows found for cache table 'storage.interface', Columns:

It goes on to list the 6 pairs of duplicate entries.  It then says it is complete:

2014-05-16 16:28:43,241 DEBUG [com.netapp.wfa.common.stats.MethodInvocationStatsInterceptor] (Thread-9484 (HornetQ-client-global-threads-1319482573)) COMPLETED

The "aquire" process runs for a total of 5 seconds.  While it is a small lab environment, that seem short which makes me think it may not be updating the WFA Cache_DB.

Any idea how to re-create the storage.interface table to remove the duplicate entries?

[korns · Accepted solution]

Interesting. I'm not sure it is related but a quick answer to 'how to re-create the xxx.table':

To reset the WFA Cache Database view for that OCUM server is:

• Execution tab
• Select Data Sources
• Click to highlight the OCUM Data Source in question
• Right-click and pick Edit
• Under Schedular configuration highlight the single line for the 'storage' scheme
• Hit the [Reset Scheme] button in lower left

That is not something you normally need to do routinely but it can't hurt and may clear the issue you're seeing.

[ag]

Hi David,

I would like to help you out.

Can you do the following for me:

1. Can you check if the dictionary object "quota" and "qtree" have "Acquisition enabled" set to true.

2. What version of WFA are you running?

3. Run an acquisition cycle from WFA

4. Attach the following in your response

    a. Please navigate here:

     <path_to_WFA_installation>\jboss\standalone\tmp\wfa

       Take a snapshot of this folder.

     b. send the server.log file which contains information of the latest acquisition cycle.

[DavidSpano]

Anil,

I am un-sure how to check the "Acquisition enabled" for "quota" and "qtree".  Here is what I did in WFA console:

Designer > Cache Queries > selected "Schema:storage, Dictionary Entry:Qtree > Right-click>Edit

The "disk_limit_mb" attribute is set (checked) for "To be Cached".  The same is true for storage.quota Attrbutes.

We are running WFA 2.2.0.2.4RC1; build number: 2298704

I am new to Communities, so as soon as I figure out how to attache the log files to this discussion I will send the additional information.

Thanks.

PS.  Request logs are attached.

[adaikkap]

Hi David,

Change tio the advanced mode in the right corner of the reply post.

That would allow you to attach files.

Regards

adai

Sent from my NEXUS 7, excuse typos.

[DavidSpano]

Anil,

I zipped up "<path_to_WFA_installation>\jboss\standalone\tmp\wfa" files, but I could not get the zip file attached.

However, I did notice that the temporary files for quota and qtree had the new/correct disk_limit_mb values:

localhost-storage.qtree-2737410280991564570

197 lab_test_qtree1 Warning \N 156 unix 1200.00000 0 0.00000

localhost-storage.quota-1445606745540657310

197 TREE ladisk01:/lab_common/lab_test_qtree1 0.00000 1200.00000 0 156 197

Even so, the WFA database still has the old value of 5000.00

[ag]

David,

You can see if a dictionary object has Acquisition Enabled by navigating to Designer -> Dictionary.

There is a field called 'Acquisition Enabled' for each of the dictionary objects.

Since you say that the temp file contains latest information, i will have to look at your DB.

Information is fetched from the data source and stored in these temp files from where it is prepared to be inserted into the DB.

Something must be going wrong while preparing it for insertion.

I will need a backup of your WFA cache.

From the WFA console, go to  Administration -> Backup & Restore.

When you hit 'Backup' , you get an sql.gz file.

You can attach it here as explained by Adai.

I will be able to reproduce the issue better if i have the DFM DB as well

-Anil

[adaikkap]

Hi David,

     At this time, I would recommend you to open a support case with NetApp, using your WFA System ID and ASUP log along with a controller Serial number.

I wouldn't recommend posting a WFA cache DB, on the communities.

Regards

adai

[DavidSpano]

Anil,

I was able to check the Dictionary Objects for Qtree and Quota and yes, they both have "Acquisition Enabled" set to true.

Thansk.

David

[ag]

David,

Please go ahead and raise a support case as suggested by adai.

Thanks,

Anil

[DavidSpano]

Anil,

I performed the "Reset Schema" as suggested by David Korns above and that seems to have resolved the problem.

There is what he worte:

To reset the WFA Cache Database view for that OCUM server is:

• Execution tab
• Select Data Sources
• Click to highlight the OCUM Data Source in question
• Right-click and pick Edit
• Under Schedular configuration highlight the single line for the 'storage' scheme
• Hit the [Reset Scheme] button in lower left