topic Re: Purging the DFM Database in Active IQ Unified Manager Discussions

Purging the DFM Database

kofchur — Thu, 05 Jun 2025 06:44:00 GMT

I have a customer that has been experiencing alot performance problems with reporting with his Operations Manager server (4.0.2). After doing some investigating, I came across this in tr-3440, OnCommand 5.0 Sizing Guide:

<<When the difference between the following commands are more than 2x then contact NGS to prune your db for improving the responce times.

<<dfm volume list -a and the volume list is 3x or more

<<dfm qtree list -a and qtree list is 2x or more

<<dfm lun list -a and lun list is 2x or more

<<dfm host list -a and host list is 2x or more

Going by the 4.0 Sizing Guide (the older tr-3440) it says that the max number of volumes and qtree for a setup with DFM, ProtMgr, and ProvMgr is 1650 volumes and 5800 qtrees. This customer of mine also has Performance Advisor running and the out put of the above commands is 85k+ for volumes (51x over limit) and 48k+ for qtrees (8x over limit)! So, it appears that the experienced performance problems, to at least some extent, are due to the massive ammounts of historical data lingering around and thus should be purged.

Now, I know of all the warnings about purging a filer object and that purging a filer object will also purge every record that it owns and hence could take a long time and you would be better off just installing the database. However, since I'm not intending to purge the filer object but just volumes and qtrees, what would be the impact of this? Will it take a long time and/or will it impact performance?

Also, if reinstalling the database would be the fastest and best way, is there a way to export the ProtMgr configs and reimport them back into the newly installed database? Otherwise we would have to document all the datasets and manually reconfigure them (yuck!).

Thanks for any input!

todd

Purging the DFM Database

adaikkap — Sat, 08 Oct 2011 07:37:28 GMT

The number you are talking about 1650 & 5800 volumes and qtrees respectively is for a DFM with all license enabled(ie Ops-Mgr, Perf Advisor,Prot Mgr and Prov Mgr) which is just 40 nodes.

But the number is 10K and 50K volumes and qtrees for a just a DFM server doing only OM + PA with 250 node.

If your difference between dfm volume list and dfm volume list -a is more than 3X and similarly for others, then its time you purge you data of those deleted instances from the DB.

As anyways you wont be able to access them unless you guys uses the -a options in the cli. It makes perfect sense to run the db pruning.

You should also thing about moving to OnCommand 5.0 which is a 65 bit architecture and takes advantage of the available compute and memory.

Pls do open a NGS case and get your db cleaned and perf data too, after that you will definitely feel the difference. BTW do you have lot of snapmanagers in your environment ?

If thats the case its the flexclone volumes/luns that is the cause for lot of these deleted instance.

Regards

adai

Re: Purging the DFM Database

mark_schuren — Mon, 21 Nov 2011 15:52:41 GMT

I have the same problem with OC 5.0 (upgraded over time from several previous DFM versions).

My deleted object count is also very high - yes it's Snapmanagers and their temporary vol clones / lun clones that fills up the database.

Is there an official way to prune the DB from deleted objects?

Something like "DELETE FROM volumes WHERE Deleted = Yes" ?

I'd like to prune the DB of our own lab DFM - not a production customer environment...

Re: Purging the DFM Database

hadrian — Wed, 30 Nov 2011 00:49:47 GMT

The official way to prune the DB from deleted objects is to open a support case with NetApp Global Support.

I've been researching it for days now to see if there is any way to avoid opening the case, but there isn't =).

Hope this helps,

Hadrian

Re: Purging the DFM Database

brothanb1 — Thu, 01 Dec 2011 17:21:56 GMT

I also would like an official way to prune the database without having to open a support case.

Re: Purging the DFM Database

mark_schuren — Thu, 01 Dec 2011 17:38:45 GMT

Ok, can we talk about the inoffcial way...? just a couple of SQL statements?

As I said it's meant for lab environment only.

Re: Purging the DFM Database

hadrian — Thu, 01 Dec 2011 23:32:42 GMT

I looked for days with no luck. The solution is to send the deleted object count over to NGS and work with them via webex to get it done.

Hadrian

Re: Purging the DFM Database

kofchur — Wed, 11 Jan 2012 19:05:40 GMT

Adai,

This whole issue of having all these artifact objects is due to Snapmanager. Now I have almost 100k total volume objects total, both managed and "deleted," 53k qtrees and about 6000 luns. We migrated off SMO, but we still have SMVI, which as slowed down the growth, but I still need to migrate to a new DFM instance (hence my other posts that you have been responding to).

Anyway, I would strongly urge the OnCommand development team to integrate some intelligence into DFM to ignore these clones that are created in the backup process of Snapmanager so large environments like mine don't run into this again.

I ran a test on our lab of purging 600 volume objects and reloading the database. It took 50 minutes do do this. With as many objects that i would have to purge in production, this would take forever.

I opened a case with NGS and after looking at my situation, they agreed that the best course was to just migrate to a new DFM instance.

--todd

Re: Purging the DFM Database

adaikkap — Wed, 06 Mar 2013 04:46:21 GMT

We have this purge tool today that take care of cleanup all this. Dfmpurge which will remove all these stale instances but requires down time. The utility has 2 modes and gives the estimation of downtime required as well. In most cases it shouldn't take more than 30mins to cleanup.

Pls take a look at this video ( 3.43 Mins) and read the KB on how this tool works.

Video Link: DFM Purge Tool: How to Video

KB link: https://kb.netapp.com/support/index?page=content&id=1014077

Link to tool chest: http://support.netapp.com/NOW/download/tools/dfmpurge/

Regards

adai

Re: Purging the DFM Database

francoisbnc — Wed, 06 Mar 2013 15:02:44 GMT

Hi Adai,

The tools looking for these kind of objects

Qtree 13658 ( 99%) 13573 ( 99%)

Volume 9018 ( 99%) 8966 ( 99%)

Interface 32 (100%) 32 (100%)

FCP Target 17 (100%) 17 (100%)

Aggregate 4 (100%) 4 (100%)

What's about events table?, in our case pretty hudge

WHERE eventDeleted IS NULL

"COUNT(events.eventStatus)"

"12158538"

WHERE eventDeleted IS NOT NULL

"COUNT(events.eventStatus)"

"36293"

Re: Purging the DFM Database

adaikkap — Wed, 06 Mar 2013 15:19:53 GMT

Hi Francois,

As clearly said in the KB article this tool only deletes Mark deleted objects. The events pruning that you are looking for is coming in OCUM 5.2 which is currently in BETA. In 5.2 we purge the mark deleted objects as well as events. This will happen both during upgrade to 5.2 as well add every time you restore your db.

Re: Purging the DFM Database

francoisbnc — Thu, 07 Mar 2013 10:35:46 GMT

I really expect this version and hope sincerely major bugs will be fixed this time

francois

Re: Purging the DFM Database

mark_schuren — Thu, 07 Mar 2013 11:02:32 GMT

So in 5.2 the purging is done automatically during and DB restore ONLY?

Why is that not possible online during normal operations? An option like 'purge deleted objects automatically after xx weeks' would be very much appreciated...

Mark

-- sent from mobile device, please excuse any typos

-- Mark Schuren, Ultra Consulting Network GmbH

Am 06.03.2013 um 16:23 schrieb "Adaikkappan Arumugam" <xdl-communities@communities.netapp.com<mailto:xdl-communities@communities.netapp.com>>:

<https://communities.netapp.com/index.jspa>

Re: Purging the DFM Database

created by Adaikkappan Arumugam<https://communities.netapp.com/people/adaikkap> in OnCommand Management Software - View the full discussion<https://communities.netapp.com/message/102089#102089>

Hi Francois,

Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/102089#102089>

Start a new discussion in OnCommand Management Software by email<mailto:discussions-community-products_and_solutions-storage_management_software@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2026>

Re: Purging the DFM Database

adaikkap — Thu, 07 Mar 2013 12:23:18 GMT

Hi Mark

That's right in OCUM 5.2 which is currently in BETA OnCommand Unified Manager 5.2 Beta Program its only done during upgrade to or restore in 5.2.

Yes the online thing should be possible but not sure about the release its targeted.

Regards

adai

Re: Purging the DFM Database

adaikkap — Thu, 07 Mar 2013 12:24:16 GMT

Hi Francois,

Is there a list or specific bugs that you are looking for ? If so can you share them ?

Regards

adai

Re: Purging the DFM Database

francoisbnc — Thu, 14 Mar 2013 01:49:29 GMT

Hi Adai,

Sorry for the delay.

Yes I do.

- Provisioning more flexible, with compression , vol autosize on SAN prov, snap autodelete for NAS and SAN as well.

- Very slow NMC after reaching 250 Datasets, even with VM 12GB RAM 6 CPUs

- Some snapshots not purged / deleted on secondary and primary

- Events and database not purged correctely. (probably fixed in 5.2)

- Conform multiple Dataset in one action

- Cannot provision NAS Dataset with volume imported or added manually

- Current space breakout display wrong (LUN Information on NAS prov dataset)

Cheers,

francois

Re: Purging the DFM Database

adaikkap — Sun, 17 Mar 2013 07:39:30 GMT

Hi Francois,

- Provisioning more flexible, with compression , vol autosize on SAN prov, snap autodelete for NAS and SAN as well.

As you know for NAS we allow Autogrow and Auto Delete to be indepedentaly slected starting OCUm 5.1. There are no changes in OCUM 5.2. You can check the same in the BETA Pages

https://communities.netapp.com/docs/DOC-22967

- Some snapshots not purged / deleted on secondary and primary

Can you raise a case, seems like an issue.

- Events and database not purged correctely. (probably fixed in 5.2)

Yes.

- Conform multiple Dataset in one action

You should be able to script it using cli or linux one liners

- Cannot provision NAS Dataset with volume imported or added manually

Yes, this is not there in 5.2 as well.

- Current space breakout display wrong (LUN Information on NAS prov dataset)

IIRC this is fixed in OCUM 5.1

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=493743

Regards

adai

Re: Purging the DFM Database

francoisbnc — Sun, 24 Mar 2013 08:14:37 GMT

Hi Adai,

For SAN and NAS provisioning there are not other choices that commitment=destroy for snapshoots autodelete.

That is a big problem for customers that have snapmirror and snapvault relations on slow link, because some base snapshots are delete when volumes become full and that takes weeks to do an initial base line for 500GB-1TB volumes.

Regards,

francois

Re: Purging the DFM Database

clilescapario — Mon, 03 Jun 2013 15:46:01 GMT

This "feature" has always boggled my mind, PM is managing the mirrors/vaults but the provisioning side will happily shoot the protection side in the foot and remove the base snaps. I understand it's to keep the volume online, but it would be nice if there were other options for commitment.

Re: Purging the DFM Database

adaikkap — Thu, 06 Jun 2013 22:07:26 GMT

Can you open a support case with NetApp for the same ?

Regards

adai