Active IQ Unified Manager Discussions

Oncommand deleted object still in the database

f_duranti
10,572 Views

Hi all, checking on DFM database I see that there are really old data into it that have the "deleted" flag setup but are still into the database. There are also really old objects that don't exist anymore and in my view can be safely deleted (old storage that don't exist anymore or old volumes).

I see that there are sometimes really many of those items.... doing a dfm volume list -a return 7700 objects (volumes) while doing a dfm volume list return only 700.

My question is: can I delete those objects or they're needed by DFM in some way? Should they be automatically cleaned by DFM after a time period or they'll remain in the database indefinitely?

This is an example of some aggregate objects:

1009 itromfs22:aggr1-2008-11-04 12:07:37.000-1 AggregateunknownYes  dfmmonitor
1010 itromfs22:aggr1-2008-11-04 12:07:37.000-1.snapshot AggregateunknownYes  dfmmonitor
8212 itromfs22:aggr1-2010-03-08 00:04:49.000-1 AggregateunknownYes  dfmmonitor
8213 itromfs22:aggr1-2010-03-08 00:04:49.000-1.snapshot AggregateunknownYes  dfmmonitor
22234 itromfs22:aggr1.snapshot             Aggregate32_bit No                

Regards

Francesco

1 ACCEPTED SOLUTION

hondasuschi
8,980 Views

May this help

Solution

To avoid this behavior, when removing/deleting a volume from a dataset and DFM, remove it from the database as well. This is because DFM will assume that a volume deleted (in DFM) could be re-added later and could retain the historical data.
Execute the following commands to delete the Protection Manager volume permanently:
WARNING: Deleting Objects from DFM in this manner will remove them forever and will not retain historical data about the volume.
  1. dfm volume list -a (to get the IDs of the desired volumes)
  2. dfm service stop
  3. dfm service start sql
  4. dfm volume delete -f {ID of volume}
  5. dfm service start

View solution in original post

21 REPLIES 21

arunchak
10,244 Views

HI,

They will remain in the database. For now we don’t have any mechanism where DFM deletes them after a certain period of time.

Also see this post https://communities.netapp.com/message/69742#69742.

Thanks,

Arun

From: c-xdl-communities

Sent: Wednesday, January 18, 2012 11:49 PM

To: Chakravarthy, Arun

Subject: - Oncommand deleted object still in the database

<https://communities.netapp.com/index.jspa>

Oncommand deleted object still in the database

created by Francesco Duranti<https://communities.netapp.com/people/f.duranti> in OnCommand Mgmt Software - View the full discussion<https://communities.netapp.com/message/72020#72020>

Hi all, checking on DFM database I see that there are really old data into it that have the "deleted" flag setup but are still into the database. There are also really old objects that don't exist anymore and in my view can be safely deleted (old storage that don't exist anymore or old volumes).

I see that there are sometimes really many of those items.... doing a dfm volume list -a return 7700 objects (volumes) while doing a dfm volume list return only 700.

My question is: can I delete those objects or they're needed by DFM in some way? Should they be automatically cleaned by DFM after a time period or they'll remain in the database indefinitely?

This is an example of some aggregate objects:

1009 itromfs22:aggr1-2008-11-04 12:07:37.000-1 Aggregate

unknown

Yes

dfmmonitor

1010 itromfs22:aggr1-2008-11-04 12:07:37.000-1.snapshot Aggregate

unknown

Yes

dfmmonitor

8212 itromfs22:aggr1-2010-03-08 00:04:49.000-1 Aggregate

unknown

Yes

dfmmonitor

8213 itromfs22:aggr1-2010-03-08 00:04:49.000-1.snapshot Aggregate

unknown

Yes

dfmmonitor

22234 itromfs22:aggr1.snapshot

Aggregate

32_bit

No

Regards

Francesco

Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/72020#72020>

Start a new discussion in OnCommand Mgmt Software by email<mailto:discussions-community-products_and_solutions-storage_management_software@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2026>

agireesh
9,937 Views

Hi Francesco,

When you delete any volume from dfm using "dfm volume delete <volume-name-or-id>" command then that volume doesn't delete permanently from dfm database, only dfm stop monitoring that volumes.

So, the volumes those have deleted flag mark as "Yes", that means those volumes are deleted using "dfm volume delete <volume-name-or-id> command and dfm is not monitoring those volumes", so, dfm volume list -a command list those volumes because those volumes are not permanently deleted from dfm database.

So, "dfm volume list -a"  command will list all the volumes present in dfm database while "dfm volume list" command will list only those volume for which Deleted flag marked as "No" and in other word, dfm volume list

command will list only those volume which are monitor by dfm server.

To delete the objects permanently from dfm database, please refer the post link that Arun mention in his post.

Regards,

Gireesh

f_duranti
10,244 Views

Just a question about it... it's safe to clear those "old" data related to volume/aggregate that don't exist anymore in the database using the "delete -f" or they are in some way used by DFM for some kind of historical report?

arunchak ha scritto:

HI,

They will remain in the database. For now we don’t have any mechanism where DFM deletes them after a certain period of time.

Also see this post https://communities.netapp.com/message/69742#69742.

Thanks,

  Arun

arunchak
10,246 Views

HI Fransisco,

AFAIK they are not used by DFM. Deleted objects are not monitored so I don't think it has to do anything with reporting.

I would like to tell you one scenario where I use the deleted object:

Say, I have a LUN and I would want to stop monitoring it so would have deleted the LUN from dfm "dfm lun delete <id>". Now the LUN is marked deleted in DFM and gets listed under -a option.

After a month, I want to monitor the LUN again and want to add the same object in DFM, then I do the following (use dfm lun add)

[root@shoemaker-rhel6x64-01 ~]# dfm lun list -a | grep -i 1532

1532 vserver01_arun:/lun_07_vol/lun07                                   Yes     (object deleted is YES)

[root@shoemaker-rhel6x64-01 ~]# dfm lun add 1532 (Adding back the LUN to enable monitoring on it)

Added LUN vserver01_arun:/lun_07_vol/lun07 (1532).

[root@shoemaker-rhel6x64-01 ~]# dfm lun list  | grep -i 1532

1532 vserver01_arun:/lun_07_vol/lun07                                   No (Added LUN is now showing as not deleted)

[root@shoemaker-rhel6x64-01 ~]# dfm lun add help

NAME

    add -- re-monitor deleted LUNs

SYNOPSIS

    dfm lun add { all | <objects> ... }

Hope this is useful.

-Arun

agireesh
10,244 Views

Hi Francesco,

Once you delete any aggregate/Volume using -f option then all the history data for that volume or aggregate will get deleted and you can not retrieve that data  after this.

So, this is not safe operation.

Regards,

Gireesh

arunchak
10,244 Views

I would say, I agree with Gireesh, and hence we do not recommend this to the customer. If eventually the db size increases considerably due to the deleted objects customers should contact NetApp support for pruning the db.

There was a post regarding this in the community.

kjag
10,244 Views

The deleted objects are not at all used in any of the reports as its in deleted state. But you can't retrive the history data for the deleted objects anymore, once deleted using -f.

If you really dont want those objects, you can delete them from DB.

-KJag

agireesh
10,244 Views

The use case of this option (-f option ) is, suppose you deleted some object from storage system but still dfm data base store the records for that object some period of time.

So, in this case you can delete such object forcefully from dfm database using -f option.

Regards,

Gireesh

kjag
10,244 Views

Yes. You can delete the objects using "dfm volume delete -f <Volume Id>" command. You need to restart your dfm services after running this command.

-KJag

hondasuschi
8,981 Views

May this help

Solution

To avoid this behavior, when removing/deleting a volume from a dataset and DFM, remove it from the database as well. This is because DFM will assume that a volume deleted (in DFM) could be re-added later and could retain the historical data.
Execute the following commands to delete the Protection Manager volume permanently:
WARNING: Deleting Objects from DFM in this manner will remove them forever and will not retain historical data about the volume.
  1. dfm volume list -a (to get the IDs of the desired volumes)
  2. dfm service stop
  3. dfm service start sql
  4. dfm volume delete -f {ID of volume}
  5. dfm service start

adaikkap
8,675 Views

Hi Francesco,

     What problem is it causing in your environment. This is internal to OnCommand, this should not affect any of the functionality pls let us know if its otherwise.

Also as kjag said, its not used in any reports or in calculation.

regards

adai

f_duranti
8,675 Views

There's no real problem it's just that it seems that DFM server is a bit slow right now when I work with dataset and volumes and I was thinking it could be because of all the old objects not used anymore but that are in the database. The server is a vm with 8 gb of memory and 4 vcpu and sometimes it takes also a minutes to add volumes to a dataset. We have 8 storage controller defined and about 700 real volumes managed from them.

adaikkap
8,675 Views

Is your DFM server also doing protection related activity ?

what are the values of the following options ?

[root@ ~]# dfbm options list statusUpdateInterval

Option               Value

-------------------- ------------------------------

statusUpdateInterval 2 minutes

[root@ ~]# dfbm options list purgeJobsOlderThan

Option             Value

------------------ ------------------------------

purgeJobsOlderThan 12.86 weeks

Regards

adai

f_duranti
8,675 Views

Yes, it's doing it. It seems I cannot find the first option ... probably i can lower the 12 weeks..

[root@itromlx13 ~]# dfbm options list statusUpdateInterval

Error: There is no statusUpdateInterval option.

[root@itromlx13 ~]# dfbm options list purgeJobsOlderThan

Option             Value                        

------------------ ------------------------------

purgeJobsOlderThan 12.86 weeks

[root@itromlx13 ~]# dfbm options list

Option                       Value                        

---------------------------- ------------------------------

backupDirMonInterval         8 hours

backupScriptRunAs           

discoverNdmp                 Disabled

ndmpMonInterval              30 minutes

purgeJobsOlderThan           12.86 weeks

snapvaultLagErrorThreshold   1 day,  2:00

snapvaultLagWarningThreshold 1 day,  1:00

adaikkap
8,675 Views

Hi Francesco,

     These two options will keep  the db from growing in size due to the job progress message and retention of old jobs.

The first option needs to be set before able to be listed.

dfbm options set statusUpdateInterval=5min

This will populate the job progress message in protection jobs once every 5min instead of the default 30seconds.

dfbm options list purgeJobsOlderThan

This will purge all jobs older than the days specified. By this we will reduce the size of the db and you should get some better response times. Btw what version of dfm are you running ?

Regards

adai

f_duranti
7,287 Views

Thanks adai it's a good information. 5 minutes seems a good interval to put on instead of 30 seconds but does this have any impact? I've Snapcreator using pm to do remote backup and I've saw that it checks the status every minute, will it find that the mirror finished only 5 minutes after (in the worst case) or the job completion is reported directly?

I'm running 5.0d1 and I've just now discovered that dfm database backup is failing from 2 month (never checked it unfortunately) because the validatedb after backup is going wrong and it seems something related ti Host Services.

Replicating the dbvalid command that dfm use I got this massage (that don't appear in the dbbackup.log:

VALIDATE TABLE "DFM"."hsNotifications"

Primary key for "hsNotifications" has missing index entries

Foreign key "hostServiceId" has missing index entries

SQL error (-300) -- Run time SQL error -- Validation of table "hsNotifications" has failed

Any ideas? I've opened a support call but if someone had the same problem it will be nice to solve it faster:)

Francesco

adaikkap
7,286 Views

Hi Francesco,

                  The 5minutes is only for progress events, which gives information like snapmirror.progress and snapvault.progress and not for other messages of jobs. Basically this give how much data is transferred.

Can you get the output of dfm backup create -v ?

Regards

adai

f_duranti
7,286 Views

Those are the logs:

[root@itromlx13 ~]# dfm backup create -v

Disabling Performance Advisor.

DestinationPath = /opt/dfm_backup/

TempPath        = /opt/dfm_backup/dfm_temp_dest_dir/

InstallPath     = /opt/NTAPdfm/

DataPath        = /opt/NTAPdfm/data/

LogPath         = /opt/NTAPdfm/data/

KeysPath        = /opt/NTAPdfm/conf/keys/

PerfPath        = /opt/NTAPdfm/perfdata/

ScriptPath      = /opt/NTAPdfm/script-plugins/

RptPath        = /opt/NTAPdfm/reports/repository/custom/

DBFileName      = monitordb.db

LogFileName     = monitordb.log

KeysFileName    = dfmenc.keys

ArchivePath     = /opt/dfm_backup/dfm_backup_2012-01-23_17-25-05.ndb

ArchiveTempName = tmp.backup.file.ndb

strDBFilePath= /opt/dfm_backup/dfm_temp_dest_dir/monitordb.db

strDBLogFilePath= /opt/dfm_backup/dfm_temp_dest_dir/monitordb.log

strKeysFilePath= /opt/dfm_backup/dfm_temp_dest_dir/dfmenc.keys

Copying keystore for backup ...

SQL Anywhere Backup Utility Version 10.0.1.4213

(107134 of 107120 pages, 100% complete)

Database backup completed

Enabling Performance Advisor.

Error: Database validation returned errors.

Please see dfmbackup.log file for details.

This is the dfmbackup.log:

Jan 23 17:25:32 [dfm: INFO]: [23221:0x2b8da1a67080]: dbeng9 process spawned with pid = 23377.

Jan 23 17:25:32 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23378.

I. 01/23 17:25:32. SQL Anywhere Personal Server Version 10.0.1.4213

I. 01/23 17:25:32.

I. 01/23 17:25:32. Copyright (c) 2011, iAnywhere Solutions, Inc.

I. 01/23 17:25:32. Portions copyright (c) 2011, Sybase, Inc.

I. 01/23 17:25:32. All rights reserved. All unpublished rights reserved.

I. 01/23 17:25:32. 

I. 01/23 17:25:32. This software contains confidential and trade secret information of

I. 01/23 17:25:32. iAnywhere Solutions, Inc.

I. 01/23 17:25:32. Use, duplication or disclosure of the software and documentation

I. 01/23 17:25:32. by the U.S. Government is subject to restrictions set forth in a license

I. 01/23 17:25:32. agreement between the Government and iAnywhere Solutions, Inc. or

I. 01/23 17:25:32. other written agreement specifying the Government's rights to use the

I. 01/23 17:25:32. software and any applicable FAR provisions, for example, FAR 52.227-19.

I. 01/23 17:25:32. 

I. 01/23 17:25:32. iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA

I. 01/23 17:25:32.

I. 01/23 17:25:33. 4 logical processor(s) on 2 physical processor(s) detected.

I. 01/23 17:25:33. This server is licensed to:

I. 01/23 17:25:33.     DFM User

I. 01/23 17:25:33.     NetApp

I. 01/23 17:25:33. Running Linux 2.6.18-274.7.1.el5 #1 SMP Mon Oct 17 11:57:14 EDT 2011 on X86_64

I. 01/23 17:25:33. Server built for X86_64 processor architecture

I. 01/23 17:25:33. Authenticated Server licensed for use with Authenticated Applications only

I. 01/23 17:25:33. 904832K of memory used for caching

I. 01/23 17:25:33. Minimum cache size: 8192K, maximum cache size: 4087256K

I. 01/23 17:25:33. Using a maximum page size of 8192 bytes

I. 01/23 17:25:33. Starting database "dfmBackupDbvalidDatabase" (/opt/dfm_backup/dfm_temp_dest_dir//monitordb.db) at Mon Jan 23 2012 17:25

I. 01/23 17:25:33. Database recovery in progress

I. 01/23 17:25:33.     Last checkpoint at Mon Jan 23 2012 17:25

I. 01/23 17:25:33.     Checkpoint log...

Jan 23 17:25:33 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23394.

Jan 23 17:25:34 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23397.

I. 01/23 17:25:35.     Transaction log: /opt/dfm_backup/dfm_temp_dest_dir//monitordb.log...

Jan 23 17:25:35 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23400.

Jan 23 17:25:36 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23407.

I. 01/23 17:25:37.     Rollback log...

I. 01/23 17:25:37.     Checkpointing...

I. 01/23 17:25:37. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. Recovery complete

I. 01/23 17:25:37. Database "dfmBackupDbvalidDatabase" (monitordb.db) started at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. This database is licensed for use with:

I. 01/23 17:25:37.     Application: DataFabric Manager

I. 01/23 17:25:37.     Company: NetApp

I. 01/23 17:25:37. Database server started at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. Trying to start SharedMemory link ...

I. 01/23 17:25:37.     SharedMemory link started successfully

I. 01/23 17:25:37. Trying to start TDS (TCPIP) link ...

I. 01/23 17:25:37. Unable to start on default port; starting on port 49152 instead

I. 01/23 17:25:37.     TDS (TCPIP) link started successfully

I. 01/23 17:25:37. Now accepting requests

Jan 23 17:25:38 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23433.

I. 01/23 17:25:38. Cache size adjusted to 849288K

I. 01/23 17:25:42. Cache size adjusted to 836256K

I. 01/23 17:25:42. Cache size adjusted to 833672K

I. 01/23 17:25:48. Cache size adjusted to 833064K

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

SQL Anywhere Stop Server Utility Version 10.0.1.4213

I. 01/23 17:30:46. Database server shutdown requested by DBSTOP

I. 01/23 17:30:46. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:30

I. 01/23 17:30:46. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:30

Jan 23 17:30:46 [dfm: INFO]: [23221:0x2b8da1a67080]: dbstop process returned successfully.

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

Please see dfmbackup.log file for details.

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

Please see dfmbackup.log file for details.

I. 01/23 17:30:46. Database server stopped at Mon Jan 23 2012 17:30

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database backup create failed

pedro_rocha
7,286 Views

Hello,

Did you manage to correct the DFM DB backup problem?

Regards,

Pedro

adaikkap
6,089 Views

We have this purge tool today that take care of cleanup all this. Dfmpurge which will remove all these stale instances but requires down time. The utility has 2 modes and gives the estimation of downtime required as well. In most cases it shouldn't take more than 30mins to cleanup.

Pls take a look at this video ( 3.43 Mins) and read the KB on how this tool works.

Video Link: DFM Purge Tool: How to Video

KB link:  https://kb.netapp.com/support/index?page=content&id=1014077

Link to tool chest: http://support.netapp.com/NOW/download/tools/dfmpurge/

Regards

adai

Public