SnapCreator 3.6.0 & Lotus Notes 8.5.3

[CENDRE_CEDRIC]

Hi all,

I try to use SnapCreator 3.6.0 with Lotus Domino 8.5.3FP3 on a Windows Server 2008R2. All software versions are 64bit.

My NetApp Storage is FAS3220 with DOT 8.1.2 7-Mode.

Everything works correctly until taking snapshot, I get an error on the "Application unquiesce."

Here is the error log from the server SnapCreator:

########## Application unquiesce ##########

76 [Thu Feb 21 09:32:46 2013] ERROR: 500 Status read failed: An existing connection was forcibly closed by the remote host. at /<C:\Program Files\NetApp\NetApp_Snap_Creator_Framework\scServer3.6.0\snapcreator.exe>SnapCreator/Agent/Remote.pm line 541

77

78 [Thu Feb 21 09:32:46 2013] [spmail004:9090(3.6.0.1)] ERROR: [scf-00054] Application unquiesce for plugin domino failed with exit code 1, Exiting!

79

80 ########## Application unquiesce ##########

Here is the error log ( event log) from the client SnapCreator:

Application Log:

A device or program has requested attention. Device or application: C:\Program Files\IBM\Lotus\Domino\nrunjava.EXE. Message title: ERROR.

A device or program has requested attention. Device or application: C:\Program Files\IBM\Lotus\Domino\nserver.exe. Message title: C:\Program Files\IBM\Lotus\Domino\nserver.exe.

System Log:

Application popup: Microsoft Visual C++ Debug Library : Debug Error!

Program: ...pp\NetApp_Snap_Creator_Framework\scAgent3.6.0\snapcreator.exe

Invalid allocation size: 18446744073709551610 bytes.

(Press Retry to debug the application)

Do you have any idea?

Thank you for your help.

Sincerely,

Cedric.

[sivar]

could you please set this in your config file and try this again?

SC_AGENT_UNQUIESCE_TIMEOUT=1800

Looks like unquiescing takes long time.

Siva Ramanathan

Snap Creator Community Moderator

[CENDRE_CEDRIC]

Hello Siva,

I just applied this parameter, and no change.

SnapCreator & Domino crash always after taking snapshot.

The Domino server is almost empty, there is only one mailbox.

62 ########## Creating snapshot(s) ##########

63

64 ########## SNAPSHOT CREATE COMMANDS ##########

65 [Mon Feb 25 09:18:58 2013] INFO: Running snapshot create command NTAP_SNAPSHOT_CREATE_CMD01 [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:]

66 [Mon Feb 25 09:19:06 2013] INFO: Running snapshot create command [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:] finished successfully

67

68 ########## SNAPSHOT CREATE COMMANDS FINISHED SUCCESSFULLY ##########

69

70 ########## PRE APPLICATION UNQUIESCE COMMANDS ##########

71 [Mon Feb 25 09:19:06 2013] INFO: No commands defined

72

73 ########## PRE APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

74

75 ########## Application unquiesce ##########

76 [Mon Feb 25 09:19:44 2013] ERROR: 500 Server closed connection without sending any data back at /<C:\Program Files\NetApp\NetApp_Snap_Creator_Framework\scServer3.6.0\snapcreator.exe>SnapCreator/Agent/Remote.pm line 541

77

78 [Mon Feb 25 09:19:44 2013] [spmail003:9090(3.6.0.1)] ERROR: [scf-00054] Application unquiesce for plugin domino failed with exit code 1, Exiting!

79

80 ########## Application unquiesce ##########

And in the events logs :

Information 25/02/2013 09:19:47 Interactive Services detection 1000 None A device or program has requested attention. Device or application: C:\Program Files\IBM\Lotus\Domino\nserver.exe. Message title: C:\Program Files\IBM\Lotus\Domino\nserver.exe.
Information 25/02/2013 09:19:09 Interactive Services detection 1000 None A device or program has requested attention. Device or application: C:\Program Files\IBM\Lotus\Domino\nrunjava.EXE. Message title: ERROR.
Information 25/02/2013 09:19:06 Application Popup 26 None Application popup: Microsoft Visual C++ Debug Library : Debug Error!
Information 25/02/2013 09:19:05 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_TRANSLOG) was successfully created.
Information 25/02/2013 09:19:05 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_DAOS) was successfully created.
Information 25/02/2013 09:19:04 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_MAIL3) was successfully created.
Information 25/02/2013 09:19:04 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_MAIL2) was successfully created.
Information 25/02/2013 09:19:04 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_MAIL1) was successfully created.
Information 25/02/2013 09:19:04 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_CHANGEINFO) was successfully created.
Information 25/02/2013 09:19:03 SnapDrive 190 Snapshot Copy event Snapshot (spmail003_lotus-daily_20130225091838) of LUN(s) on storage system (spsda001) volume (LOTUS_SDA001_SPMAIL003_DATA) was successfully created.
Information 25/02/2013 09:19:00 SnapDrive 213 Snapshot Copy event SnapDrive is ready to create Snapshot copy (spmail003_lotus-daily_20130225091838) of LUN(s).

########## Creating snapshot(s) ##########

########## SNAPSHOT CREATE COMMANDS ##########
[Mon Feb 25 09:18:58 2013] INFO: Running snapshot create command NTAP_SNAPSHOT_CREATE_CMD01 [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:]
[Mon Feb 25 09:18:58 2013] [spmail003:9090(3.6.0.1)] DEBUG: Executing command [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:]
[Mon Feb 25 09:19:06 2013] [spmail003:9090(3.6.0.1)] TRACE: Command [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:] finished with
exit code: [0]
stdout: [Initializing...SPMAIL003 : Checking policies
SPMAIL003 : Checking access control
SPMAIL003 : Preparing LUNs for Snapshot creation
SPMAIL003 : Ready to create snapshot copy
SPMAIL003 : Creating a Snapshot for the LUN
SPMAIL003 : Initiating snapmirror update for any source volumes.
The operation completed successfully.
]
stderr: []
[Mon Feb 25 09:19:06 2013] [spmail003:9090(3.6.0.1)] DEBUG: Command [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:] finished successfully with message
[Initializing...SPMAIL003 : Checking policies
SPMAIL003 : Checking access control
SPMAIL003 : Preparing LUNs for Snapshot creation
SPMAIL003 : Ready to create snapshot copy
SPMAIL003 : Creating a Snapshot for the LUN
SPMAIL003 : Initiating snapmirror update for any source volumes.
The operation completed successfully.
]
[Mon Feb 25 09:19:06 2013] INFO: Running snapshot create command [sdcli snap create -s spmail003_lotus-daily_20130225091838 -D G: E: I: J: K: O: T:] finished successfully

########## SNAPSHOT CREATE COMMANDS FINISHED SUCCESSFULLY ##########

########## PRE APPLICATION UNQUIESCE COMMANDS ##########
[Mon Feb 25 09:19:06 2013] INFO: No commands defined

########## PRE APPLICATION UNQUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## Application unquiesce ##########
[Mon Feb 25 09:19:44 2013] ERROR: 500 Server closed connection without sending any data back at /<C:\Program Files\NetApp\NetApp_Snap_Creator_Framework\scServer3.6.0\snapcreator.exe>SnapCreator/Agent/Remote.pm line 541

[Mon Feb 25 09:19:44 2013] [spmail003:9090(3.6.0.1)] ERROR: [scf-00054] Application unquiesce for plugin domino failed with exit code 1, Exiting!

########## Application unquiesce ##########

Do you have another idea ?

Thanks for tour help,

Cedric.

[sivar]

Please request a NetApp Global Support case for this issue.

This might need further troubleshooting from our engineers.

Also, is this a VMware implementation? Are there RDM LUNs involved?

[CENDRE_CEDRIC]

Hello Siva,

There are 10 days ago, I opened a NetApp Case (P2 Level), and this time no return, from where my post on the community.

Absolutely, I use a type infrstructure Vmware ESXi 5.1 ( iSCSI datastores).

My Lotus-Domino Vms also use iSCSI luns via NetApp SnapDrive mode.

My Lotus-Domino is working in snapcreatoragent mode.

I have a master server to control the backup snapcreator for my 4 lotus-domino systems.

I followed the following Technicals Reports:

TR-3588  / TR-3756 / TR-3843 / TR-3883 / TR-3917 / TR-4009.

Best Regards,

Cedric.

[sivar]

Thank you for the detail.

I am checking with my contacts.

1) Meanwhile, please supply me the Domino logs.

     C:\Program Files\IBM\Lotus\Domino\data\ibm_technical_support\*.log

2) What Antivirus solution is used in these servers? are there any antivirus logs that could shed some light on this issue?

Thanks,
Siva Ramanathan

[sivar]

You can email me the logs here sivar at netapp.com

[sivar]

Hello,

All my research points to the DAOS feature.

From the console.log, I found this information below.

07FC:0002-034C] 25/02/2013 15:32:00   Informational - The DAOS catalog is not synchronized. Deletions will be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize.

And also,

[0C8C:0009-0F6C] 25/02/2013 15:33:10   RM Evaluation: Started RMLogicalDisk

[0C8C:000A-0F88] 25/02/2013 15:33:11   Disk Spindle Information was not provided in the Directory

[0C8C:0009-0F6C] 25/02/2013 15:33:11   RM Evaluation: Started RMMemory

[0C8C:0009-0F6C] 25/02/2013 15:33:11   RM Evaluation: Started RMNetwork

[0C8C:0009-0F6C] 25/02/2013 15:33:11   RM Evaluation: Started RMProcessor

[0C8C:0009-0F6C] 25/02/2013 15:33:11   RM Evaluation: Initialization complete

[0C8C:0007-0CC8] 25/02/2013 15:35:31   ISpy: An error occurred getting ISpy's inbox view

[0C8C:0007-0CC8] 25/02/2013 15:35:31   ISpy: Mail Poller shutting down due to a fatal error.

I will reach out to my contacts on the "Disk Spindle information error", and keep you posted on the progress.

[CENDRE_CEDRIC · Accepted solution]

Hi everybody,

I solved this issue.

I just changed in my Lotus/Domino configuration, the Domino logical links ( link to folder ) by a simply Windows mount points.

And now, it works perfectly ( snapshot / unqiesce / ... ).

Thank you to all those who have helped me.

Best Regards,

Cedric.

[PIYUSHBANSAL198722]

I am facing the same issue but using citrix xen plugin.

Could you please elaborate as to what exactly did you do to resolve the issue??

I am unable to understand the steps...