Domino Plugin does not clear transaction logs

[SVOLLRAT1]

Hi All

We have a serious Problem with one of our customers Domino Instances.

Yesterday the Agent suddenly wasn't reachable anymore and after several Server and Domino restarts we are at least at the Point where Jobs are running green from SnapCreator point of view.

One major issue still remains, the Translogs won't be truncated and moved to ChangeInfo Directory.

The debug log only shows one Message related to that and it is not very telling:

Archiving of transaction logs already in progress.Error begin archiving logs.

Any Ideas what may hang here? Could it be something caused by SnapCreator or is that some deadlock in Domino that needs to be cleared?

Log-Partition is 90% full and I expect customer escalation within the next view hours.

Thanks in advance for your help

BR

Stefan

[ktenzer]

Hi Stefan,

I have forwarded this to our Domino Expert John Spinks but he wont be in for another few hours, he is on EST.

The error is coming from Domino from what I can tell and so it would appear that there is a hung archiving command within the domino system and that domino isnt allowing a new one to start. I would call IBM and ask if there is a way to manually clear this and explain the situation. I would think they can help.

Snap Creator should ideally be able to clear or cancel the hung archiving job but I am not sure this is possible, I would have to ask engineers if IBM provides mechanism to do this. Nevertheless that isnt something that will help for this situation but we will look into this and see of we can prevent this situation in future.

Keith

[SVOLLRAT1]

Thanks Keith

Just what I expected to be honest.

Hope John can later add some of his experience to this so that I can further push it towards IBM.

BR Stefan

[ktenzer]

Yeah I completely understand what situation you are in and I feel aweful about this, I have been there too

IBM probably will say well the backup application in this case (SnapCreator) is at fault and they need to fix things.

John has connection to IBM engineering so I think once he comes on we can have a real conversation with them and try to find a solution to fix this problem. I would assume somewhere IBM had a command or something you can do to cancel a running archive (which in this case for whatever reason is hung). I cant imagine they dont have a capability to do that or at least a procedure.

As for Snap Creator, assuming there is an API to cancel a running archive, we can definitely do it from our end but this would require a code change. Again we will get the team looking into this.

Do you have capability to add more storage to log volume? If so I would go that route to buy more time

Keith

[SVOLLRAT1]

System is owned and operated by Customer, for us it is virtually a BlackBox, only thing we have there is the SC-Agent and more or less are responsible for all that may or may not fault because of Backup.

So Customer has to do or request the Disk-Extension, if they won't there is no thing I could do about it.

[spinks · Accepted solution]

Stefan,

As Keith mentioned this error is coming from Domino itself - Snap Creator is just reporting it.

A quick search shows a similar error message on Symantec's site.  They are likely using the same APIs that we are.

The link is:  http://www.symantec.com/business/support/index?page=content&id=TECH47910

The solution is to stop Domino and verify that ALL Domino related processes have stopped.

Especially verify that logasio is stopped as that handles the log movement.

Once everything is stopped, start Domino back up. 

Then run Snap Creator.

If that doesn't work we need to engage IBM.

Any idea what the original issue was?  You mentioned that you already restarted both the server and Domino several times to get this far.

Thanks,

John

[SVOLLRAT1]

Hi John

Thanks for the great hint, forwarded it to the Customer.

I don’t know what the initial Issue was, unfortunately the Logs got purged in the doing.

But it is well possible that the Logger-Process faulted right from the start and customer didn’t properly clear the processes, I only have their word for what they did or didn’t do.

Freundliche Grüsse / Kind Regards

Stefan Vollrath

T-Systems Schweiz AG

Storage & Backup Operations

Stefan Vollrath

Storage Engineer

Murgenthalstrasse 12, CH-4901 Langenthal

+41 (0) 78 645 1076 (phone)

E-Mail: stefan.vollrath@t-systems.com

http://www.t-systems.ch<http://www.t-systems.ch/>

Notice: This transmittal and/or attachments may be privileged or confidential. If you are not the intended recipient, you are hereby notified that you have received this transmittal in error; any review, dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you.

Von: John Spinks

Gesendet: Freitag, 7. Februar 2014 13:45

An: Vollrath, Stefan

Betreff: - Re: Domino Plugin does not clear transaction logs

<https://communities.netapp.com/index.jspa>

Re: Domino Plugin does not clear transaction logs

created by John Spinks<https://communities.netapp.com/people/spinks> in Snap Creator - Plugins - View the full discussion<https://communities.netapp.com/message/125115#125115>

Stefan,

As Keith mentioned this error is coming from Domino itself - Snap Creator is just reporting it.

A quick search shows a similar error message on Symantec's site. They are likely using the same APIs that we are.

The link is: http://www.symantec.com/business/support/index?page=content&id=TECH47910

The solution is to stop Domino and verify that ALL Domino related processes have stopped.

Especially verify that logasio is stopped as that handles the log movement.

Once everything is stopped, start Domino back up.

Then run Snap Creator.

If that doesn't work we need to engage IBM.

Any idea what the original issue was? You mentioned that you already restarted both the server and Domino several times to get this far.

Thanks,

John

Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/125115#125115>

Start a new discussion in Snap Creator - Plugins by email<mailto:discussions-community-products_and_solutions-databases_and_enterprise_apps-snapcreator-plugins@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2539>

[spinks]

Any updates? 

Did you get this working?

Thanks!

[SVOLLRAT1]

Hi John

Customer finally realized that it in fact was the Archiver hanging within Domino.

Freundliche Grüsse / Kind Regards

Stefan Vollrath

T-Systems Schweiz AG

Storage & Backup Operations

Stefan Vollrath

Storage Engineer

Murgenthalstrasse 12, CH-4901 Langenthal

+41 (0) 78 645 1076 (phone)

E-Mail: stefan.vollrath@t-systems.com

http://www.t-systems.ch<http://www.t-systems.ch/>

Notice: This transmittal and/or attachments may be privileged or confidential. If you are not the intended recipient, you are hereby notified that you have received this transmittal in error; any review, dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you.

Von: John Spinks

Gesendet: Freitag, 14. Februar 2014 03:11

An: Vollrath, Stefan

Betreff: - Re: Domino Plugin does not clear transaction logs

<https://communities.netapp.com/index.jspa>

Re: Domino Plugin does not clear transaction logs

created by John Spinks<https://communities.netapp.com/people/spinks> in Snap Creator - Plugins - View the full discussion<https://communities.netapp.com/message/125527#125527>

Any updates?

Did you get this working?

Thanks!

Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/125527#125527>

Start a new discussion in Snap Creator - Plugins by email<mailto:discussions-community-products_and_solutions-databases_and_enterprise_apps-snapcreator-plugins@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2539>