Workflows from OCUM Fail

[amayacitta]

Hey guys,

When I do something simple from OCUM like a snapmirror update I get errors, it looks like HTTPS is unable to connect.

If i connect to the windows box where WFA is installed I can connect to those systems fine. Also under settings,  datasources they appear fine.

23:31:57.054 INFO [Transfer SnapMirror] ### Command 'Transfer SnapMirror' in 'PERL' ###
23:31:59.742 INFO [Transfer SnapMirror] Connecting to the cluster: DC3PNETAPP01/DestinationClusterIp
23:31:59.805 INFO [Transfer SnapMirror] Credentials successfully provided for 'DC3PNETAPP01.ad.xxxxxx.co.uk'
23:31:59.867 INFO [Transfer SnapMirror] Credentials successfully provided for 'DC3PNETAPP01.ad.xxxxxx.co.uk'
23:31:59.930 INFO [Transfer SnapMirror] Trying to connect to DC3PNETAPP01.ad.xxxxxx.co.uk using HTTPS on port 443 with timeout 60000
23:32:00.023 INFO [Transfer SnapMirror] Error while connecting HTTPSin NaServer::verify_server_certificate: server certificate verification failed: unable to get local issuer certificate
23:32:00.086 INFO [Transfer SnapMirror] Trying to connect to DC3PNETAPP01.ad.xxxxxx.co.uk using HTTP on port 80 with timeout 60000
23:32:01.195 INFO [Transfer SnapMirror] Error while connecting HTTPSin Zapi::invoke, cannot connect to socket
23:32:01.242 ERROR [Transfer SnapMirror] Command failed for Workflow 'Transfer SnapMirror relationship' with error :
Unable to connect to array

23:32:01.242 INFO [Transfer SnapMirror] ***** Workflow Execution Failed ***** 

If I run this test script, it works fine. I followed this forum post and configured the environmental variables. I can run the perl script from any directory, so they work fine.

I'm lost anyone got any ideas?

####

use NaServer;

#Set Array credentials below
$array="DC1PNETAPP01.ad.xxxxxxxx.co.uk";
$username = "xxxxxxxx";
$pass = "xxxxxxxx";

$server = NaServer->new($array, 1, 14);
$server->set_style('LOGIN');
$server->set_admin_user($username, $pass);
$server->set_transport_type('HTTPS');
$server->set_server_cert_verification(0);
$output = $server->invoke( "system-get-version" );
if (ref ($output) eq "NaElement" && $output->results_errno != 0) {
# failed to establish connection
print "Connection failed";

print $output->results_reason;
}
else {
# successfully connected
print "Connected successfully";
}

########

[amayacitta]

This bit makes me wonder about certificate trusts, how do I add them? they are trusted from the windows side, I've also added the roots into the Java certificate store. Do they need to go somewhere else as well? Maybe perl has a certificate root store too that doesnt use the windows one?

23:32:00.023 INFO [Transfer SnapMirror] Error while connecting HTTPSin NaServer::verify_server_certificate: server certificate verification failed: unable to get local issuer certificate

Perhaps this but I dont fully understand it..

http://docs.activestate.com/activeperl/5.26/perl/lib/LWP/Protocol/https.html

"If hostname verification is requested by LWP::UserAgent's ssl_opts, and neither SSL_ca_file nor SSL_ca_path is set, then SSL_ca_file is implied to be the one provided by Mozilla::CA. If the Mozilla::CA module isn't available SSL requests will fail. Either install this module, set up an alternative SSL_ca_file or disable hostname verification.

This module used to be bundled with the libwww-perl, but it was unbundled in v6.02 in order to be able to declare its dependencies properly for the CPAN tool-chain. Applications that need https support can just declare their dependency on LWP::Protocol::https and will no longer need to know what underlying modules to install."

[mbeattie]

Hi,

I tested the WFA certified PERL command "transfer snapmirror" on WFA 5.0.1 succuessfully in my lab. Is it possible that there is a certifcate issue on the destination vserver? Have you tried cloning the command and disabling hostname verification? EG

$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
my $wfa_util = WFAUtil->new();

Do you get the same error when updating the snapmirror relationship on a different vserver? Have you checked the certification expiration? EG

cluster2::> certificate show -fields expiration -type server -vserver vserver1_dr
  (security certificate show)
vserver     common-name serial           ca          type   subtype cert-name                    expiration
----------- ----------- ---------------- ----------- ------ ------- ---------------------------- ------------------------
vserver1_dr vserver1_dr 15B61346913114F7 vserver1_dr server -       vserver1_dr_15B61346913114F7 Wed Jul 29 13:58:34 2020

/Matt

[amayacitta]

Hi Matt,

I'm running 5.1.3212, the certificate hasn't expired and works fine on the WFA host outside of perl, I assume there is a way to import trusted root certificates and intermediates into the perl environment. Do you know how to do this? I've imported them into windows and Java  but this doesn't effect the perl runtime.

Happy to try disabling SSL verify once but I dont fancy doing that for all workflows, doing it once will at least help us understand if it is or isnt SSL that's the issue.

[amayacitta]

Seems this is how it's done, just trying to work out what is meant and if I can pernamently add a CA rather than doing it per workflow. If anyone has done this before let me know.

https://metacpan.org/pod/LWP::UserAgent#ssl_opts

[amayacitta]

OK so its 100% certificate based with Perl, if I set this to 0 then it works fine

$server->set_server_cert_verification(1);

I tried adding the root and subordinate certs to the PEM file for Mozilla as part of perl, which is in "C:\Program Files\NetApp\WFA\Perl64\lib\Mozilla\CA". This hasn't helped, I'm assuming there is another part to this to get the CA certs into the Perl runtime.

The WFA host itself has no issues connecting to the cluster management LIF cert, it's all fine. So it's 100% the Perl runtime and it's ability to accept the cert, what I dont understand is under normal conditions you would have self assigned certificates on the cluster managment LIF - I fail to see how this works out the box. Perhaps this can be taken up with engineering or a higher level? Is it possible something has changed in regards to certificates in the latest versions on WFA and Perl, I'm running the following version of Perl with 5.1.3212 Work Flow Automation.

PS C:\Windows\system32> perl -version

This is perl 5, version 26, subversion 3 (v5.26.3) built for MSWin32-x64-multi-thread
(with 2 registered patches, see perl -V for more detail)

Copyright 1987-2018, Larry Wall

Binary build 2603 [e95bda1d] NetApp provided by ActiveState http://www.ActiveState.com
Built May 29 2019 22:07:02

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

[paul_stejskal]

If WFA is failing and you've root caused the effect, then you can open a case. I don't work with the OnCommand products so a case is the best option.

[mjdalton1]

What do you get when you run this on your clusters?  Source and destination...

security certificate show -vserver * -common-name clustername*

Looks like a local certificate has expired.

[paul_stejskal]

Also what version of ONTAP?

[amayacitta]

The certs are all valid until 2029, they all work fine outside of the Perl runtime - via windows (on the WFA server) for example. This effects all 5 clusters, the versions are either 9.5P9 or 9.6P2.

The certs are issued by internal CA's which are trusted and chained OK. If there were issues on the NetApp end with the certificates it would effect more than just the Perl runtime.

DC3PNETAPP01::> security certificate show -vserver * -common-name DC3P*
Vserver    Serial Number   Certificate Name                       Type
---------- --------------- -------------------------------------- ------------
DC3PNETAPP01 
           1A0000000F7XXXXXXXXXX4000000000000F 
                           DC3NETAPP01                            server
    Certificate Authority: XXX-Sub02
          Expiration Date: Tue Dec 11 12:02:59 2029

I guess it's time to log a call, will sort Monday.