OCUM 7.2P1 Backup Failing - Error occurred while copying files inside temporary directory

[Chris_Jones]

I have a freshly deployed OCUM 7.2P1 Virtual Appliance. The database backups are enabled and backing up to the default location, /data/ocum-backup/

In the OCUM events I see this event every day:

Daily database backup 'UM_7.2P1_backup_unix_11-07-2017-18-00.7z' creation failed. Reason: Copying a file or directory is failed due to a file system error.

I enabled the diag user and had a look at the /var/log/ocum/ocum-error.log and it has these entries at 6pm every day:

2017-11-06 18:00:15,309 ERROR [oncommand] [taskScheduler-3] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupProperties] Error occurred while copying files inside temporary directory in Backup path
2017-11-06 18:00:15,405 ERROR [oncommand] [taskScheduler-3] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Error while creating the backup

I read this NetApp KB, https://kb.netapp.com/support/s/article/internal-um-database-backup-errors, that mentions the permissions on the file:

/opt/netapp/essentials/rp/security/proxy.server.p12

I've checked my virtual appliance at the /opt/netapp/essentials/rp/security folder is empty.

Any thoughts on what else I can check?

Thanks,

Chris

[Chris_Jones]

I've also checked the OCUM debugs and I see this every day:

2017-11-07 18:00:14,716 ERROR [oncommand] [taskScheduler-3] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupProperties] Error occurred while copying files inside temporary directory in Backup path
java.nio.file.AccessDeniedException: /data/ocie/server.keystore
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[na:na]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:na]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:na]
        at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:230) ~[na:na]
        at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:581) ~[na:na]
        at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253) ~[na:na]
        at java.nio.file.Files.copy(Files.java:1274) ~[na:1.8.0_144]
        at com.netapp.dfm.impl.backup.BackupProperties.copyToTempDirectory(BackupProperties.java:176) ~[dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupRestoreManager.createBackupInTransaction(BackupRestoreManager.java:223) [dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupRestoreManager.createBackup(BackupRestoreManager.java:185) [dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupRestoreManager$$FastClassBySpringCGLIB$$7bcdd65f.invoke(<generated>) [dfm-impl.jar:7.2P1]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core.jar:4.3.4.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651) [spring-aop.jar:4.3.4.RELEASE]
        at com.netapp.dfm.impl.backup.BackupRestoreManager$$EnhancerBySpringCGLIB$$466e19c4.createBackup(<generated>) [dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupScheduleJob.run(BackupScheduleJob.java:23) [dfm-impl.jar:7.2P1]
        at com.netapp.dfm.core.scheduling.DelegatingJob.execute(DelegatingJob.java:53) [dfm-app-common.jar:7.2P1]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz.jar:na]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context.jar:4.3.4.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_144]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
        at com.netapp.dfm.common.logging.MDCClearingThreadFactory$1.run(MDCClearingThreadFactory.java:28) [dfm-common.jar:7.2P1]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
2017-11-07 18:00:14,781 ERROR [oncommand] [taskScheduler-3] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Error while creating the backup
com.netapp.dfm.impl.backup.BackupCreationException: Copying a file or directory is failed due to a file system error.
        at com.netapp.dfm.impl.backup.BackupProperties.copyToTempDirectory(BackupProperties.java:180) ~[dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupRestoreManager.createBackupInTransaction(BackupRestoreManager.java:223) ~[dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupRestoreManager.createBackup(BackupRestoreManager.java:185) ~[dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupRestoreManager$$FastClassBySpringCGLIB$$7bcdd65f.invoke(<generated>) ~[dfm-impl.jar:7.2P1]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core.jar:4.3.4.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651) ~[spring-aop.jar:4.3.4.RELEASE]
        at com.netapp.dfm.impl.backup.BackupRestoreManager$$EnhancerBySpringCGLIB$$466e19c4.createBackup(<generated>) ~[dfm-impl.jar:7.2P1]
        at com.netapp.dfm.impl.backup.BackupScheduleJob.run(BackupScheduleJob.java:23) ~[dfm-impl.jar:7.2P1]
        at com.netapp.dfm.core.scheduling.DelegatingJob.execute(DelegatingJob.java:53) [dfm-app-common.jar:7.2P1]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz.jar:na]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context.jar:4.3.4.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_144]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
        at com.netapp.dfm.common.logging.MDCClearingThreadFactory$1.run(MDCClearingThreadFactory.java:28) [dfm-common.jar:7.2P1]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.nio.file.AccessDeniedException: /data/ocie/server.keystore
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[na:na]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:na]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:na]
        at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:230) ~[na:na]
        at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:581) ~[na:na]
        at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253) ~[na:na]
        at java.nio.file.Files.copy(Files.java:1274) ~[na:1.8.0_144]
        at com.netapp.dfm.impl.backup.BackupProperties.copyToTempDirectory(BackupProperties.java:176) ~[dfm-impl.jar:7.2P1]
        ... 18 common frames omitted

[SYNTAXERROR · Accepted solution]

Hi Chris

Go into the diag console mode.

Check the permissions on the file:

ls -al  /data/ocie/server.keystore

and change it:

chown jboss:root /data/ocie/server.keystore

I hope this helps as I had the same error in a customer environment..

Kind regards

Dario

[Chris_Jones]

Hi Dario,

Thanks for the reply, good to know it's not just me 🙂

I tried changing ownership of the file but got this error:

diag@svr-ocum:/data/ocie$ chown jboss:root /data/ocie/server.keystore
chown: changing ownership of `/data/ocie/server.keystore': Operation not permitted

I had to run this command:

sudo chown jboss:root /data/ocie/server.keystore

I changed the database backup schedule to run in a couple of minutes and it succeeded. Thank you so much for this, really appreciated.

I am deploying OCUM 7.2P1 at a customer site early next week and am happy I was able to resolve this before being onsite.

Thanks for this!

Chris

[SYNTAXERROR]

Hi Chris

You're welcome. 😃

Sure I always do a

sudo su -

at the beginning so I forgot to tell you that.

Good to hear that it worked.

Cheers

Dario