DFM backups fail after upgrade to 4.0.2

cecil · ‎2012-07-09

Does anyone have any insight on this error?

dfm backup create -v
Disabling Performance Advisor.
DestinationPath = /share/dfm_db_backups/
TempPath        = /share/dfm_db_backups/dfm_temp_dest_dir/
InstallPath     = /opt/NTAPdfm/
DataPath        = /opt/NTAPdfm/data/
LogPath         = /opt/NTAPdfm/data/
KeysPath        = /opt/NTAPdfm/conf/keys/
PerfPath        = /opt/NTAPdfm/perfdata/
ScriptPath      = /opt/NTAPdfm/script-plugins/
RptPath        = /opt/NTAPdfm/reports/repository/custom/
DBFileName      = monitordb.db
LogFileName     = monitordb.log
KeysFileName    = dfmenc.keys
ArchivePath     = /share/dfm_db_backups/dfm_backup_2012-07-09_08-18-45.ndb
ArchiveTempName = tmp.backup.file.ndb

strDBFilePath= /share/dfm_db_backups/dfm_temp_dest_dir/monitordb.db

strDBLogFilePath= /share/dfm_db_backups/dfm_temp_dest_dir/monitordb.log

strKeysFilePath= /share/dfm_db_backups/dfm_temp_dest_dir/dfmenc.keys
Copying keystore for backup ...
SQL Anywhere Backup Utility Version 10.0.1.4213
(596331 of 596183 pages, 100% complete)
Database backup completed
Enabling Performance Advisor.
Error: dbstop process returned error.

DFMBackup.log

Jul 09 07:27:07 [dfm: INFO]: [28401:0x7ffd6bd41700]: script_dir_size=0.

Jul 09 07:34:54 [dfm: INFO]: [31452:0x7f5ca8387700]: dbeng9 process spawned with pid = 31753.

Jul 09 07:34:54 [dfm: INFO]: [31452:0x7f5ca8387700]: dbvalid process spawned successfully, pid = 31754.

I. 07/09 07:34:55. SQL Anywhere Personal Server Version 10.0.1.4213

I. 07/09 07:34:55.

I. 07/09 07:34:55. This software contains confidential and trade secret information of

I. 07/09 07:34:55. iAnywhere Solutions, Inc.

I. 07/09 07:34:55. Use, duplication or disclosure of the software and documentation

I. 07/09 07:34:55. by the U.S. Government is subject to restrictions set forth in a license

I. 07/09 07:34:55. agreement between the Government and iAnywhere Solutions, Inc. or

I. 07/09 07:34:55. other written agreement specifying the Government's rights to use the

I. 07/09 07:34:55. software and any applicable FAR provisions, for example, FAR 52.227-19.

I. 07/09 07:34:55.

I. 07/09 07:34:55. iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA

I. 07/09 07:34:55.

I. 07/09 07:34:55. 2 physical processor(s) detected.

I. 07/09 07:34:55. This server is licensed to:

I. 07/09 07:34:55. DFM User

I. 07/09 07:34:55. NetApp

I. 07/09 07:34:55. Running Linux 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 on X86_64

I. 07/09 07:34:55. Server built for X86_64 processor architecture

I. 07/09 07:34:55. Authenticated Server licensed for use with Authenticated Applications only

I. 07/09 07:34:55. 117424K of memory used for caching

I. 07/09 07:34:55. Minimum cache size: 8192K, maximum cache size: 2186536K

I. 07/09 07:34:55. Using a maximum page size of 8192 bytes

I. 07/09 07:34:55. Starting database "dfmBackupDbvalidDatabase" (/share/dfm_db_backups/dfm_temp_dest_dir//monitordb.db) at Mon Jul 09 2012 07:34

I. 07/09 07:34:55. Database recovery in progress

I. 07/09 07:34:55. Last checkpoint at Mon Jul 09 2012 07:33

I. 07/09 07:34:55. Checkpoint log...

I. 07/09 07:34:55. Transaction log: /share/dfm_db_backups/dfm_temp_dest_dir//monitordb.log...

I. 07/09 07:34:55. Rollback log...

I. 07/09 07:34:55. Checkpointing...

I. 07/09 07:34:55. Starting checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jul 09 2012 07:34

I. 07/09 07:34:55. Finished checkpoint of "dfmBackupDbvalidDatabase" (monitordb.db) at Mon Jul 09 2012 07:34

I. 07/09 07:34:55. Recovery complete

I. 07/09 07:34:55. Database "dfmBackupDbvalidDatabase" (monitordb.db) started at Mon Jul 09 2012 07:34

I. 07/09 07:34:55. This database is licensed for use with:

I. 07/09 07:34:55. Application: DataFabric Manager

I. 07/09 07:34:55. Company: NetApp

I. 07/09 07:34:55. Database server started at Mon Jul 09 2012 07:34

I. 07/09 07:34:55. Trying to start SharedMemory link ...

I. 07/09 07:34:55. SharedMemory link started successfully

I. 07/09 07:34:55. Trying to start TDS (TCPIP) link ...

I. 07/09 07:34:55. Unable to start on default port; starting on port 49152 instead

I. 07/09 07:34:55. TDS (TCPIP) link started successfully

I. 07/09 07:34:55. Now accepting requests

Jul 09 07:34:56 [dfm: INFO]: [31452:0x7f5ca8387700]: dbvalid process spawned successfully, pid = 31788.

I. 07/09 07:35:10. Cache size adjusted to 1665984K

I. 07/09 07:35:15. Cache size adjusted to 2052296K

SQL Anywhere Stop Server Utility Version 10.0.1.4213

Unable to stop database server -- Database server not found.

Jul 09 07:37:21 [dfm:ERROR]: [31452:0x7f5ca8387700]: dbstop process returned error = 1.

Jul 09 07:37:21 [dfm:ERROR]: [31452:0x7f5ca8387700]: dbstop process returned error.

Jul 09 07:37:21 [dfm:ERROR]: [31452:0x7f5ca8387700]: Database backup create failed

=====================================================

Problem started after upgrade. Have been able to do a manual backup once in awhile but all scheduled nightly backups fail.

dcornely1 · ‎2012-07-09

We need more information about your setup before the problem can be identified:

1) What kind of system is the DFM server? Windows or Linux? (assuming linux based on above but please confirm)

2) What kind of filesystem is /share/dfm_db_backups/? Is it a local filesystem? An NFS mount?

3) Can you read/write files to /share/dfm_db_backups/ outside of DFM?

Thanks.

cecil · ‎2012-07-09

Appreciate the response.

1) What kind of system is the DFM server? Windows or Linux? (assuming linux based on above but please confirm) --- SUSE Linux Enterprise Server 11 Sp1

2) What kind of filesystem is /share/dfm_db_backups/? Is it a local filesystem? An NFS mount? --- Yes, this is a NFS mount from a NetApp filer.

3) Can you read/write files to /share/dfm_db_backups/ outside of DFM? -- Yes, other Solaris hosts can read and write to this area. I also just mounted it to do double check and can read, write to that folder. This had been working for some time prior to the upgrade.

Thank you for your help.

dcornely1 · ‎2012-07-09

Okay, thanks. I ran into something similar a few weeks ago after upgrading from DFM4.0.2 to OCv5.0. My new RHEL5.6 systems had an NFS share I was trying to use for backups but also wouldn't work. The issue, in my case, was that nfslockd (rpc.statd) was not running. In the world of RHEL, this was resolved by simply running service nfslock start to start this daemon -- I know DFM backups to an NFS share does indeed work and is supported.

My suspicion is you need one of the NFS client services running that isn't, it's just a matter of identifying which one. Try this (noting that I don't have SUSE experience so you may have to tweak the commands for SUSE):

See if portmap and nfslock are configured to run:
- chkconfig --list portmap
- chkconfig --list nfslock
If the above output shows that either of these is turned off during multi-user run levels then turn it on:
- service portmap on
- service nfslock on
- chkconfig portmap on
- chkconfig nfslock on
Try another backup.

Also, by outside of DFM I meant the root user on the DFM server itself aside from the DFM application - can the root user read/write files to the NFS share? If so, this should eliminate NFS permissions as the issue.

cecil · ‎2012-07-10

First, let me apologize. The title should have said upgrade from 4.0.2, as I have upgraded to 5.0.1. I am not a SUSE guy myself but I will try an figure out what you are saying regarding portmap and nfslock. Thank you for your help! I did verify the permissions as you explained above and everything looks good. I also moved the database backup loc to the local disk (however this is a VM).

adaikkap · ‎2012-07-09

Running DFM database in a NFS mount is not a supported configuration. As sybase 10 is not supported on NFS mounts.

Regards

adai

cecil · ‎2012-07-10

Adai,

The database is located on the local disk. The DFM backup dir is on the NFS mount location. Is this supported? Regardless, I moved the backup dir to the local disk and it still fails with this same errors.

cecil · ‎2012-07-10

More Info - If I do a dfm backup create without the verify (-L) DFM will create a backup....... Not sure what that tells me though.

adaikkap · ‎2012-07-10

Hi Cecil,

Backup dir location can be on a NFS location. BTW -L is a hidden switch to skip validation of the database. A normal db backup does verification and validation, that fails in your case.But when validation is skipped it succeeds only means that there are some validation problem in the db. I strongly recommend you to open a case with NetApp Support to get your db fixed.

Regards

adai

cecil · ‎2012-07-11

Thank you both for your help...

DFM backups fail after upgrade to 4.0.2

Postgresql type cluster backup failed

Upgrade DFM 4.0.2 to OnCommand Needs Your Advise

PowerShell Toolkit Won't Upgrade

DFM download

Upgrade Advisor enhanced with better advice and more concise upgrade plans