update failed with reason: in Zapi::invoke failed to connect SSL netapp-harvest OCUM 6.3RC1

[jedrzej_lisowski]

I'm having issues with fetching capacity data from OCUM. I see following errors in netapp-harvest log:

[2015-11-24 12:00:01] [WARNING] [aggregate] update failed with reason: in Zapi::invoke failed to connect SSL
[2015-11-24 12:00:01] [WARNING] [aggregate] data-list update failed.
[2015-11-24 12:15:00] [WARNING] [aggregate] update failed with reason: in Zapi::invoke failed to connect SSL
[2015-11-24 12:15:00] [WARNING] [aggregate] data-list update failed.
[2015-11-24 12:30:00] [WARNING] [aggregate] update failed with reason: in Zapi::invoke failed to connect SSL
[2015-11-24 12:30:00] [WARNING] [aggregate] data-list update failed.

I can access OCUM via SSL without any issues. Seems that it does capture data from time to time - but fails like that most of the time.

[msaravan]

Hi,

What is the ONTAP version you are running ?

If possible, can you please create support bundle and put it in a shared path to debug it further ?

Regards,

Saravanan

[jedrzej_lisowski]

Saravanan,

It only fails to caputre from OCUM, data captures from NetApp filers work fine.

Regards

JL

[madden]

Hi,

The error “in Zapi::invoke failed to connect SSL“  comes from the SDK:

      Net::SSLeay::connect($ssl) or return $self->fail_response(13001,
           "in Zapi::invoke failed to connect SSL $!")

From that SDK code there should be a reason “$!” but none is provided making it hard to know why it aborted.

If you restart the poller does the problem go away?  

Is collection ok, then not ok, then ok again without restarting the poller?  Or is it ok, until it breaks when you start getting this error?

I think this is the fault of some intermediate module (either NetApp SDK or Net:: SSLeay from the internet)...

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

[jedrzej_lisowski]

Chris,

After restart, it captures data once or twice, and then it stops. Sometimes restart helps, sometimes not. Seems that it can capture some data, and can't some. I do have all the dedupe and compression savings data, I don't seem to get used aggregate data. After it breaks it doesn't seem to come back.

Regards

JL

[madden]

Hi JL,

I think the issue might be related to SSL libraries in use or those libraries trying to failback to something less secure than TLS (which isn't supported by OCUM or cDOT) for some reason.

Can you check/upgrade to the latest packages related to SSL?

--perl-Net-SSLeay

--openssl

Can you run harvest from a different host platform (so different linux box or distro, or from windows) to see if the behavior changes?  

Are you sure the network is error free between the poller and OCUM?

Can you try a more frequent poll interval and see if you have any change in behavior?

Cheers,

Chris

[jedrzej_lisowski]

Chris,

I'm running that on Ubuntu 15.10 (64-bit), Network is error free between those hosts - i'm 100% sure.

Versions:

ii openssl 1.0.2d-0ubuntu1 amd64 Secure Sockets Layer toolkit - cryptographic utility

ii libnet-ssleay-perl 1.70-1 amd64 Perl module for Secure Sockets Layer (SSL)

I changed from 15 minutes to 10 and errors are still there.

I will try running harvest from a different host (maybe Centos 7.x)

[jedrzej_lisowski]

Chris,

I've moved netapp-harvest to centos 7, i've changed pooling time to 5 minutes, and I can still notice those issues:

[2015-11-26 10:25:01] [WARNING] [aggregate] update failed with reason: in Zapi::invoke failed to connect SSL
[2015-11-26 10:25:01] [WARNING] [aggregate] data-list update failed.
[2015-11-26 10:30:05] [WARNING] [lun] update failed with reason: in Zapi::invoke failed to connect SSL
[2015-11-26 10:30:05] [WARNING] [lun] data-list update failed.
[2015-11-26 10:35:10] [WARNING] [qtree] update failed with reason: in Zapi::invoke failed to connect SSL
[2015-11-26 10:35:10] [WARNING] [qtree] data-list update failed.

versions:

openssl-1.0.1e-42.el7.9.x86_64
perl-Net-SSLeay-1.55-3.el7.x86_64

Could that be caused by the amount of data it has to capture from OCUM?