Solved: Problem with dfm Backup

f_duranti · ‎2012-02-14

Hi, I'm opening a new post on this problem I had. Our backup is going bad from some times, I've a support call opened but it seems that it takes ages to get an answer so I'm trying on the forum to see If I'm a bit more lucky here

I've manually replicated the procedure that backup do to get the real error message (it's not in the log so it seems difficult to understand why backup are getting wrong).

Being able to do the validate table on the manually copied and started up database i get this erro:

VALIDATE TABLE "DFM"."hsNotifications"

Primary key for "hsNotifications" has missing index entries

Foreign key "hostServiceId" has missing index entries

SQL error (-300) -- Run time SQL error -- Validation of table "hsNotifications" has failed

It seems that I should only need to recreate or correct those 2 tables.

I'm almost sure the problem was with some older data inside those table related to when I had onCommand Host 1.0 on a server that is no more inside dfm (it uninstalled not in a really good way).

The real problem is that there's no dfm hs delete -f so I'm not able to delete those stale entry.

THose are the dfm hs list, dfm hs list -a, dfm hs list -av. As you can see there are 2 hosts that are still inside the tables but that I've no way to remove and I'm sure the problem is there.

Anyone can help me with this?

[root@itromlx13 ~]# dfm hs list

Id Host Name Host Address Version Status Timezone

---------- ---------------------------------------- -------------------- ---------- ---------------------- ----------------------------------------

40155 ITNAPIS10.Q8INT.COM 10.14.5.10 1.1.0.1512 up GMT-1:00(1 hour East of UTC).

41243 ITROMIS10.Q8INT.COM 10.160.5.110 1.1.0.1512 up GMT-1:00(1 hour East of UTC).

[root@itromlx13 ~]# dfm hs list -a

Id Host Name Host Address Version Status Timezone Deleted Deleted By

---------- ---------------------------------------- -------------------- ---------- ---------------------- ---------------------------------------- ---------- ----------

40155 ITNAPIS10.Q8INT.COM 10.14.5.10 1.1.0.1512 up GMT-1:00(1 hour East of UTC). No

41243 ITROMIS10.Q8INT.COM 10.160.5.110 1.1.0.1512 up GMT-1:00(1 hour East of UTC). No

35248 ITROMIS100.Q8INT.com 10.160.5.200 1.1.0.1512 down GMT-1:00(1 hour East of UTC). Yes dfmserver

35067 ITROMIS60.Q8INT.com 10.160.5.60 1.0.0.1493 down GMT-2:00(2 hours East of UTC). Yes dfmserver

[root@itromlx13 ~]# dfm hs list -va

Id Host Name Host Address Version Status Timezone Deleted Deleted By

---------- ---------------------------------------- -------------------- ---------- ---------------------- ---------------------------------------- ---------- ----------

Id: 40155

Host Name: ITNAPIS10.Q8INT.COM

Host Address: 10.14.5.10

Admin Port: 8699

Version: 1.1.0.1512

Management Port: 8799

Status: up

Timezone: GMT-1:00

Plugin Type: OnCommand Host Service VMware Plug-in

Plugin Version: 1.1.0.0

Id: 41243

Host Name: ITROMIS10.Q8INT.COM

Host Address: 10.160.5.110

Admin Port: 8699

Version: 1.1.0.1512

Management Port: 8799

Status: up

Timezone: GMT-1:00

Plugin Type: OnCommand Host Service VMware Plug-in

Plugin Version: 1.1.0.0

Id: 35248

Host Name: ITROMIS100.Q8INT.com

Host Address: 10.160.5.200

Admin Port: 8699

Version: 1.1.0.1512

Management Port: 8799

Status: down

Timezone: GMT-1:00

Plugin Type: OnCommand Host Service VMware Plug-in

Plugin Version: 1.0.0.0

Id: 35067

Host Name: ITROMIS60.Q8INT.com

Host Address: 10.160.5.60

Admin Port: 8699

Version: 1.0.0.1493

Management Port: 8799

Status: down

Timezone: GMT-2:00

Plugin Type: OnCommand Host Service VMware Plug-in

Plugin Version: 1.0.0.0

Those are the logs of dfm backup create -v

[root@itromlx13 ~]# dfm backup create -v

Disabling Performance Advisor.

DestinationPath = /opt/dfm_backup/

TempPath = /opt/dfm_backup/dfm_temp_dest_dir/

InstallPath = /opt/NTAPdfm/

DataPath = /opt/NTAPdfm/data/

LogPath = /opt/NTAPdfm/data/

KeysPath = /opt/NTAPdfm/conf/keys/

PerfPath = /opt/NTAPdfm/perfdata/

ScriptPath = /opt/NTAPdfm/script-plugins/

RptPath = /opt/NTAPdfm/reports/repository/custom/

DBFileName = monitordb.db

LogFileName = monitordb.log

KeysFileName = dfmenc.keys

ArchivePath = /opt/dfm_backup/dfm_backup_2012-01-23_17-25-05.ndb

ArchiveTempName = tmp.backup.file.ndb

strDBFilePath= /opt/dfm_backup/dfm_temp_dest_dir/monitordb.db

strDBLogFilePath= /opt/dfm_backup/dfm_temp_dest_dir/monitordb.log

strKeysFilePath= /opt/dfm_backup/dfm_temp_dest_dir/dfmenc.keys

Copying keystore for backup ...

SQL Anywhere Backup Utility Version 10.0.1.4213

(107134 of 107120 pages, 100% complete)

Database backup completed

Enabling Performance Advisor.

Error: Database validation returned errors.

Please see dfmbackup.log file for details.

This is the dfmbackup.log:

Jan 23 17:25:32 [dfm: INFO]: [23221:0x2b8da1a67080]: dbeng9 process spawned with pid = 23377.

Jan 23 17:25:32 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23378.

I. 01/23 17:25:32. SQL Anywhere Personal Server Version 10.0.1.4213

I. 01/23 17:25:32.

I. 01/23 17:25:32. This software contains confidential and trade secret information of

I. 01/23 17:25:32. iAnywhere Solutions, Inc.

I. 01/23 17:25:32. Use, duplication or disclosure of the software and documentation

I. 01/23 17:25:32. by the U.S. Government is subject to restrictions set forth in a license

I. 01/23 17:25:32. agreement between the Government and iAnywhere Solutions, Inc. or

I. 01/23 17:25:32. other written agreement specifying the Government's rights to use the

I. 01/23 17:25:32. software and any applicable FAR provisions, for example, FAR 52.227-19.

I. 01/23 17:25:32.

I. 01/23 17:25:32. iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA

I. 01/23 17:25:32.

I. 01/23 17:25:33. 4 logical processor(s) on 2 physical processor(s) detected.

I. 01/23 17:25:33. This server is licensed to:

I. 01/23 17:25:33. DFM User

I. 01/23 17:25:33. NetApp

I. 01/23 17:25:33. Running Linux 2.6.18-274.7.1.el5 #1 SMP Mon Oct 17 11:57:14 EDT 2011 on X86_64

I. 01/23 17:25:33. Server built for X86_64 processor architecture

I. 01/23 17:25:33. Authenticated Server licensed for use with Authenticated Applications only

I. 01/23 17:25:33. 904832K of memory used for caching

I. 01/23 17:25:33. Minimum cache size: 8192K, maximum cache size: 4087256K

I. 01/23 17:25:33. Using a maximum page size of 8192 bytes

I. 01/23 17:25:33. Starting database "dfmBackupDbvalidDatabase" (/opt/dfm_backup/dfm_temp_dest_dir//monitordb.db) at Mon Jan 23 2012 17:25

I. 01/23 17:25:33. Database recovery in progress

I. 01/23 17:25:33. Last checkpoint at Mon Jan 23 2012 17:25

I. 01/23 17:25:33. Checkpoint log...

Jan 23 17:25:33 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23394.

Jan 23 17:25:34 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23397.

I. 01/23 17:25:35. Transaction log: /opt/dfm_backup/dfm_temp_dest_dir//monitordb.log...

Jan 23 17:25:35 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23400.

Jan 23 17:25:36 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23407.

I. 01/23 17:25:37. Rollback log...

I. 01/23 17:25:37. Checkpointing...

I. 01/23 17:25:37. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. Recovery complete

I. 01/23 17:25:37. Database "dfmBackupDbvalidDatabase" (monitordb.db) started at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. This database is licensed for use with:

I. 01/23 17:25:37. Application: DataFabric Manager

I. 01/23 17:25:37. Company: NetApp

I. 01/23 17:25:37. Database server started at Mon Jan 23 2012 17:25

I. 01/23 17:25:37. Trying to start SharedMemory link ...

I. 01/23 17:25:37. SharedMemory link started successfully

I. 01/23 17:25:37. Trying to start TDS (TCPIP) link ...

I. 01/23 17:25:37. Unable to start on default port; starting on port 49152 instead

I. 01/23 17:25:37. TDS (TCPIP) link started successfully

I. 01/23 17:25:37. Now accepting requests

Jan 23 17:25:38 [dfm: INFO]: [23221:0x2b8da1a67080]: dbvalid process spawned successfully, pid = 23433.

I. 01/23 17:25:38. Cache size adjusted to 849288K

I. 01/23 17:25:42. Cache size adjusted to 836256K

I. 01/23 17:25:42. Cache size adjusted to 833672K

I. 01/23 17:25:48. Cache size adjusted to 833064K

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

SQL Anywhere Stop Server Utility Version 10.0.1.4213

I. 01/23 17:30:46. Database server shutdown requested by DBSTOP

I. 01/23 17:30:46. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:30

I. 01/23 17:30:46. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jan 23 2012 17:30

Jan 23 17:30:46 [dfm: INFO]: [23221:0x2b8da1a67080]: dbstop process returned successfully.

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

Please see dfmbackup.log file for details.

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database validation returned errors.

Please see dfmbackup.log file for details.

I. 01/23 17:30:46. Database server stopped at Mon Jan 23 2012 17:30

Jan 23 17:30:46 [dfm:ERROR]: [23221:0x2b8da1a67080]: Database backup create failed

marz · ‎2012-02-16

Greetings Francesco

Steps to resolve this problem have been delivered to the EMEA Team. They will work with you on scheduling to apply the correction on your server.

Thanks

Alex

View solution in original post

marz · ‎2012-02-15

What is the case number?

f_duranti · ‎2012-02-15

Case number is 2002833610

marz · ‎2012-02-15

Thank you. We will continue updates through the case.

marz · ‎2012-02-16

Greetings Francesco

Steps to resolve this problem have been delivered to the EMEA Team. They will work with you on scheduling to apply the correction on your server.

Thanks

Alex

f_duranti · ‎2012-02-21

Alex, thanks for your help. Support team solved the problem yesterday.

Francesco

janihujanen · ‎2012-03-30

Hi,

I Have same database backup problem with DFM. What was the resolution?

br, Jani

adaikkap · ‎2012-03-30

Raise a support case and quote the following case# to get a quick resolution 2002833610

Regards

adai

su_yue · ‎2016-07-07

can you shares this case's root cause?

very thanks