OnCommand Java connection reset errors

[DAIRYMILK]

Hi there,

Firstly, sorry for the amount of logs I'm just about to paste...

I'm having an issue with OnCommand 5.0 and the Host package 1.1, both installed on the same Windows 2008 R2 server

There is a remote vCenter server

The installation went fine and there were no errors in the installation log.  However, as soon as I added the Host package, I started getting these errors in the "server" log file under C:\program files\netapp\OnCommand Host Package\vmwareplugin\log folder.  The events are also flooding the event log with warning messages.

Error is as follows and seems to occur anytime that an object is registered or updated.  The objects are appearing in OnCommand OK, even though a warning message will be generated.  If I try to browse to the url mentioned, I get a 403 error. 

2012-03-16 12:03:11,041 [::] INFO  - Received modify event on datastore-432

2012-03-16 12:03:11,057 [::] DEBUG - Sending events : [ Event: Changed Resource:  Type: Virtualization.VMware.Datastore Name: local:esxihost01 Object Id: 4ee7752c-e74de509-811b-0025b51b004f]

2012-03-16 12:03:11,057 [::] DEBUG - Get HS client for : https://oncommandserver.domain:8799/HostServices/Mgmt

2012-03-16 12:03:11,104 [::] WARN  - Interceptor has thrown exception, unwinding now

org.apache.cxf.interceptor.Fault: Connection reset

at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:96)

at org.apache.cxf.interceptor.BareOutInterceptor.handleMessage(BareOutInterceptor.java:68)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)

at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:472)

at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:302)

at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:254)

at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)

at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:127)

at $Proxy72.notifyEvent(Unknown Source)

at com.netapp.smvi.SMConfigBundle.notifyHS(SMConfigBundle.java:142)

at com.netapp.smvi.resourcegraph.client.HsEventListener.notifyEvent(HsEventListener.java:38)

at com.netapp.smvi.resourcegraph.updater.EventWatcher.sendEvents(EventWatcher.java:408)

at com.netapp.smvi.resourcegraph.updater.EventWatcher.run(EventWatcher.java:165)

Caused by: com.ctc.wstx.exc.WstxIOException: Connection reset

at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:313)

at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:94)

... 12 more

Caused by: java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(Unknown Source)

at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)

at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)

at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)

at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1913)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1868)

at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:42)

at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)

at org.apache.cxf.io.CacheAndWriteOutputStream.write(CacheAndWriteOutputStream.java:68)

at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:96)

at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)

at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:311)

... 13 more

2012-03-16 12:03:11,104 [::] DEBUG - Could not send event to host services endpoint : Connection reset

Can anybody suggest any reasons for these?

Many thanks

[crocker]

Hi,

is this an install issue that your team handles or should this go to support?
Mike

[mharding]

I have a note into the Product team in India to get an answer for you, but if this is time-sensitive you should open a support ticket.

[DAIRYMILK]

Hi Mike/crocker,

Thanks for your responses.  I have opened a case but we have to go through third-party and they are not very fast to respond so I thought I'd try the community to see if anyone had any ideas.

If you do hear anything back from the Product team it would be great to know.

Many thanks

[adaikkap]

Hi,

     I have not seen this issue before. Issue like this are better handled by support as it may require some amount of debugging and logs to figure out whats happening.

Regards

adai

[kryan]

I installed OCHP 1.1 on a Windows 2008R2  OC 5.0 server today and although there are lots of debug level messages in the server log, none are exactly like the messages above.

[DAIRYMILK]

Thanks both for your replies.  It is a strange issue, I have a case open now so I'll make sure I post the solution when it gets resolved.

[DAIRYMILK]

So, I've had a case open for a while but all support can come up with is that they haven't seen it before and it seems to be some sort of network issue.

It would be helpful to know whereabouts the communication is breaking down.  I have the OnCommand Core and host package on the same server but I can't tell from the error message if the problem is communication between those two components or something external (maybe vCenter) and the host package.

Is anyone able to offer any further suggestions about this one?  There is a windows firewall between OnCommand and vCenter but for the purpose of resolving this issue, I have allowed all traffic and it makes no difference.

I appreciate that it must be something to do with my environment but I don't know where to start looking.  I've rebuilt the server/reinstalled OnCommand and still get the same problem but I'm not sure where the root of the issue is.

[DAIRYMILK]

Further to this, if anyone is interested, the issue appears to be related to the security baseline used when our Windows servers are built.  I tried a new server with no additional security settings and had no problems at all.  Applied the security settings (Microsoft's Security Compliance baseline) and the issue comes back.

So, now to find out which of the 200+ settings is causing the issue.....

[DAIRYMILK · Accepted solution]

OK, found the issue.  We have pretty tight security restrictions in our environment and our security officer has stipulated that we must use FIPS compliant algorithms for encryption on our member servers, enforced by group policy.

A little investigation proved that this setting is causing the problem.  I have disabled this, i.e. do not enforce only FIPS compliant algorithms, and the issue has been resolved.  I guess that the apache version used in OnCommand is not compliant.

Hope this helps anyone working in a similar environment.