SnapCreator Plugin "Cache" SoapFault

[ADVUNIBN1]

Sometimes we get the following error... after we restared the agent service everything work fine for a few days until it happens again.

Snap Creator Framework 4.0p1

### Snap Creator Framework 4.0p1

NetApp Release 8.1.2P4

########## Application Quiesce ##########

[2013-08-30 13:00:02,107] ERROR: SCF-00037: Application quiesce for plugin [CACHE] failed with error [org.apache.cxf.binding.soap.SoapFault: Can't call method "references" on an undefined value at /<C:\Program Files\NetApp\Snap_Creator_Framework\scAgent4.0p1\snapcreator.exe>SOAP/Lite.pm line 2799.

] and exit code [-1], Exiting!

Does anyone has an idea to resolv this issue?

Regards

Benny

[ktenzer]

Looks like a SOAP communication issue, under certain conditions or at times it appears the CACHE plugin returns an invalid result type back to SC server. You can try with SC 3.6P1 as this was more lenient toward SOAP message formulation, however ultimately this appears to be a plugin issue.

I will look into plugin and see if I can find anything

Regards,

Keith

[ADVUNIBN1]

Thank for reply!

We use some more Plugins and did the Update from 3.6 to 4.0 a few weeks ago. At the moment its difficult to do the downgrade.

Did you find something in the plugin?

Regards,

Benny

[sivar]

Hello Benny,

Please email a scdump output to sivar at netapp.com

I shall review the logs and get back to you asap.

Thanks,

Siva Ramanathan

[ADVUNITO1]

Hi Siva Ramanathan,

I have a similar error with the MAXDB plugin at another customer (first occurrence after customer upgraded to SC 4.0P1):

[2013-08-28 10:35:34,790] ERROR: SCF-00037: Application quiesce for plugin [maxdb] failed with error [org.apache.cxf.binding.soap.SoapFault: Can't call method "references" on an undefined value at /<C:\Program Files\NetApp\Snap_Creator_Framework\scAgent4.0p1\snapcreator.exe>SOAP/Lite.pm line 2799.

For me this looks like it is not a plugin issue but a general SOAP problem. I can mail you two SCDUMPs if you find that helpful. Just let me know if you'd like them.

Regards,

Tobias

[ADVUNITO1]

Hi,

do you have any update on this matter?

Best regards

Tobias

[sivar]

Tobias,

I don't have an update yet.

If you want me to review this with you over a webex session, I can schedule one for you.

Please email me your availability next week to sivar at netapp.com

Thanks,
Siva Ramanathan

[ADVUNIBN1]

Hi,

we have got now three issues at three customers with the same error:

[2013-08-28 10:35:34,790] ERROR: SCF-00037: Application quiesce for plugin [maxdb] failed with error [org.apache.cxf.binding.soap.SoapFault: Can't call method "references" on an undefined value at /<C:\Program Files\NetApp\Snap_Creator_Framework\scAgent4.0p1\snapcreator.exe>SOAP/Lite.pm line 2799.

] and exit code [-1], Exiting!

The error occurs at different plugins (Cache, maxDB) and always the agent chrashes.

We need assistence...

Could you find some further informations?

Regards Benjamin

[ktenzer]

BURT 761317 has been opened and this has been escalated so it is being looked at. We arent able yet to reproduce problem but hopefully soon we will and then we can provide a fix or make recommendation for workaround.

If you could provide all the environment details and steps to reproduce this error that would be helpful? Especially interesting would be if perl is installed on these systems and if so what version? Does this happen on only windows systems or unix? Is there some pattern here that can be recognized?

As for workaround: I heard from the BURT case that SC 3.6 P1 does not have this issue so if you are in a pinch and need to get something working asap I would go that route until this issue is resolved.

Regards,

Keith

[ktenzer]

Oh and another thing to try I forgot to mention is set this in config:

PLUGIN_RESULT_FILTERING_ENABLE=Y

SC 4.0 server is using cxf which is very strict with SOAP message handling and the SC Agent is using Perl which isnt a strict. This option basically will ensure the Agent filters anything out of response that plugin might be adding which shouldnt be there.

I dont know that this will fix issue but please test this, it would help us tremendously to know.

?

Thanks!

Keith

[ADVUNIBN1]

Thank for your answer!

Our issues occure alway with Windows OS.

One customer tried PLUGIN_RESULT_FILTERING_ENABLE, but the issue even exists.

I try to figure out things about perl...

[ktenzer]

Ok I wasn't too hopeful that would resolve issue but it is helpful to know that issue is outside of plugin. It sounds on paper like a compatibility issue between perl that may be installed on system and the perl we ship with Snap Creator. However in 4.0 we use same version as 3.6 so this doesnt make sense. If 4.0 doesnt work than niether should 3.6.

I am going to spend some more time trying to reproduce this tomorrow but I know where to focus at least...which is windows and maxdb.

Regards,

Keith

[ktenzer]

Ok I have reproduced the problem.

It appears to happen in windows 2008 environment R2 when running SC Agent as a service. All plugins are affected not just maxdb. I ran backups every minute for several hours and after a while you run into this issue. It only affects quiesce, other operations work strangely.

We are still trying to pinpoint the exact problem but for now you can try disabling service and running SC agent manually: scAgent4.0.0/snapcreator.exe --start-agent 9090 --verbose --debug

If this works you can setup a task manager job to run bat script which starts SC server manually upon system startup and runs it in background. Basically this is same as what service does.

I will update this when we pinpoint problem and then we should have either an acceptable workaround or a planned patch / fix.

Regards,

Keith

[ktenzer]

UPDATE:

We have identified the problem. The watchdog functionality which monitors agent is not working in some windows environments properly, windows 2008 and possibly others. The watchdog process forks a process each time a quiesce request comes into agent and starts a timer, if no unquiesce comes in the watchdog will do a force unquiesce. This is to ensure if SC server crashes that database doesnt stay in a backup state.

For windows encironments experiencing this issues please disable watchdog and things will work, you can do that by setting following in config:

SC_AGENT_WATCHDOG_ENABLE=N

This issue will be resolved in SC 4.1 with the new agent. As for SC 4.0 we are not going to fix this issue. We may release a patch which will disable watchdog for windows and throw friendly message if you try and enable it. Watchdog is not critical to agent operation and doesnt solve problem if agent crashes and database is in backup mode.

In SC 4.1 we have redesigned the agent and not only do we handle case where quiesce comes but no unquiesce happens due to SC server or network issue but also if agent crashes. The 4.1 agent will be much more robust and 4.1 will be available soon as a community release and beginning of next year as netapp supported.

Regards,

Keith

[ADVUNITO1]

Hi Keith,

thanks for the info, I have forwarded that to my customer and asked them to disable the watchdog.

I will get back to you as soon as I have received feedback.

Regards,

Tobias