Subscribe

snap creator 4.1.1 failing to quiesce domino

[ Edited ]

Hi all, looking for a little help wih getting snap creator working on one of our domino servers.


We're using the exact same configuration, snap creator version, java version, snap drive version as on other domino servers where snap creator is working.


Software versions on server

Java 7 Update 79(64 bit)
Snap Creator Framework 4.1.1
Snapdrive 7.1.1.6640

Domino Product Version 9.0.14.15158


Attached is a log file of the backup I manually started last night. Any help appreciated.


Thanks

Re: snap creator 4.1.1 failing to quiesce domino

I have already tried upgrading to snapcreator 4.1.2 to no avail.

 

Re: snap creator 4.1.1 failing to quiesce domino

It looks like your domino backup (quiesce) takes longer than 5 minutes as observed from the logs.

 

You may edit your config file and set 

SC_AGENT_TIMEOUT=3600

 

The default value is 5 minutes or 300 seconds.

Here we can set to an hour (3600 seconds) and try a backup again

 

 

Re: snap creator 4.1.1 failing to quiesce domino

Hi Sivar, thanks for the reply however I have already adjusted that threshold and this made no difference.

 

Here's what I have it set at

 

#     Agent Options                                                                                                    #
########################################################################################################################
SC_AGENT_UNQUIESCE_TIMEOUT=305
SC_AGENT_WATCHDOG_ENABLE=N
SC_AGENT=domino01.domain.co.uk:9090
SC_AGENT_LOG_ENABLE=Y
SC_AGENT_TIMEOUT=3600

Re: snap creator 4.1.1 failing to quiesce domino

Not sure how relevant this is but I noticed the wrapper.exe process does not run during the backup process

Re: snap creator 4.1.1 failing to quiesce domino

I reviewed the logs again.

########## Application Quiesce ##########
[2015-12-15 21:00:26,733] INFO: Application Quiesce for plugin : domino
[2015-12-15 21:05:45,973] ERROR: [domino01.mearsgroup.co.uk:9090 (4.1.1.1)] Operation failed. Reason: The watchdog killed the running operation. Verify that plugins/wrapper/wrapper binary has execute permission set. Increase the WRAPPER_TIMEOUT_IN_MSEC value in agent.properties if operations require more time to complete.
[2015-12-15 21:05:45,973] ERROR: [domino01.mearsgroup.co.uk:9090(4.1.1.1)] SCF-00038: Application quiesce for plug-in [domino] failed with exit code [102], continuing with backup.
[2015-12-15 21:05:46,093] INFO: Application quiesce result is empty  from plugin domino. skipping config update operation
[2015-12-15 21:05:46,093] INFO: Application Quiesce for plugin : domino finished successfully

 

Looks like exactly after 5 minutes 19 seconds a watchdog operation has killed the running operation. (backup)

 

But, interestingly your config file does not even have watchdog enabled.

 

I am thinking we can try the below

SC_AGENT_WATCHDOG_ENABLE=Y

SC_AGENT_UNQUIESCE_TIMEOUT=3605

 

Thanks,
Siva Ramanathan

 

 

Re: snap creator 4.1.1 failing to quiesce domino

correcting my own statement

 

as per

https://library.netapp.com/ecmdocs/ECMP1650312/html/GUID-382C4683-8746-406C-82C3-DF3DA782DCD2.html

 

Note: SC_AGENT_WATCHDOG_ENABLE is deprecated for use with Snap Creator Agent 4.1, and applicable only for use with Agent 4.0.

 

Hence, you may just set

SC_AGENT_UNQUIESCE_TIMEOUT=3605

Re: snap creator 4.1.1 failing to quiesce domino

Hi Sava, unquiesce is not the issue unfortunately it is the quiesce stage.

 

Here's an extract from another of our domino servers having the same issue

 

2015-12-16 13:46:38,332] INFO: STORAGE-02091: Listing Snapshot copies on volume [domino02_maildb] finished successfully.
########## Storage discovery finished successfully ! ##########
########## Pre Application Quiesce commands ##########
[2015-12-16 13:46:38,348] INFO: Pre application quiesce commands are not defined
[2015-12-16 13:46:38,349] INFO: Pre Application Quiesce commands finished successfully
[2015-12-16 13:46:38,356] INFO: Validate volume is not enabled skipping validate volume task.
########## Application Quiesce ##########
[2015-12-16 13:46:38,361] INFO: Application Quiesce for plugin : domino
[2015-12-16 13:46:44,243] ERROR: [domino02.domain.co.uk:9090 (4.1.1.1)] Operation failed. Reason: The watchdog killed the running operation. Verify that plugins/wrapper/wrapper binary has execute permission set. Increase the WRAPPER_TIMEOUT_IN_MSEC value in agent.properties if operations require more time to complete.
[2015-12-16 13:46:44,243] ERROR: [domino02.domain.co.uk:9090(4.1.1.1)] SCF-00038: Application quiesce for plug-in [domino] failed with exit code [102], continuing with backup.
[2015-12-16 13:46:44,252] INFO: Application quiesce result is empty  from plugin domino. skipping config update operation
[2015-12-16 13:46:44,252] INFO: Application Quiesce for plugin : domino finished successfully
########## Application Quiesce finished successfully ##########
########## POST APPLICATION QUIESCE COMMANDS ##########
[2015-12-16 13:46:44,254] INFO: Post application quiesce commands are not defined
########## POST APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY  ##########
########## Pre NTAP commands ##########
[2015-12-16 13:46:44,261] INFO: Pre NTAP commands are not defined
[2015-12-16 13:46:44,262] INFO: Pre NTAP commands finished successfully
########## Generating Info ASUP on fernsan04 ##########
[2015-12-16 13:46:44,375] INFO: STORAGE-01001: Creating AutoSupport message with event id [0], category [Backup Started], description [INFO: Snap Creator Framework 4.1.1 Backup for domino02 ACTION: backup POLICY: daily Plugin: domino [Supported] Volumes: domino02_snapinfo,domino02_maildb, Started], level [INFO], hostname [domino02 [domino02.domain.co.uk:9090]]
[2015-12-16 13:46:44,375] DEBUG: <ems-autosupport-log>
    <computer-name>domino02 [domino02.domain.co.uk:9090]</computer-name>
    <event-id>0</event-id>
    <event-source>SNAPCREATOR</event-source>
    <app-version>Snap Creator Framework 4.1.1</app-version>
    <category>Backup Started</category>
    <event-description>INFO: Snap Creator Framework 4.1.1 Backup for domino02 ACTION: backup POLICY: daily Plugin: domino [Supported] Volumes: domino02_snapinfo,domino02_maildb, Started</event-description>
    <log-level>6</log-level>
    <auto-support>false</auto-support>
</ems-autosupport-log>

Re: snap creator 4.1.1 failing to quiesce domino

During my internal bug search on this error, I did see a bug that seems to have been fixed with an upcoming release 5.0

 

Meanwhile, please share me the values below from your etc/agent.properties file (this file is in the agent install folder on the domino server)

OPERATION_TIMEOUT_IN_MSEC

WRAPPER_TIMEOUT_IN_MSEC

 

Thanks,
Siva Ramanathan

 

Re: snap creator 4.1.1 failing to quiesce domino

Thanks Siva, when is 5.0 due for release ?

 

Sure, here's the whole log file

 

DEFAULT_PORT=9090
KEYSTORE_FILE=etc/keystore.jks
KEYSTORE_PASS=snapcreator
CONTEXT_LIFETIME_IN_MSEC=1800000
CONTEXT_CLEANUP_FREQUENCY_IN_MSEC=30000
STORAGE_RECEIVE_TIMEOUT_IN_MSEC=600000
STORAGE_CONNECTION_TIMEOUT_IN_MSEC=60000
CATALOG_RECEIVE_TIMEOUT_IN_MSEC=60000
CATALOG_CONNECTION_TIMEOUT_IN_MSEC=30000
WATCHDOG_LISTENER_MAX_DELAY_IN_MSEC=10000
OPERATION_TIMEOUT_IN_MSEC=3600000
AUTHORIZED_HOSTS=*
THREAD_POOL_SIZE=8
RESULT_MESSAGES_LOG_LEVEL=INFO
WRAPPER_TIMEOUT_IN_MSEC=3600000
PRINT_CONFIG_VALUES_IN_LOG=false