Workflows from OCUM Fail

[amayacitta]

Hey guys,

When I do something simple from OCUM like a snapmirror update I get errors, it looks like HTTPS is unable to connect.

If i connect to the windows box where WFA is installed I can connect to those systems fine. Also under settings,  datasources they appear fine.

23:31:57.054 INFO [Transfer SnapMirror] ### Command 'Transfer SnapMirror' in 'PERL' ###
23:31:59.742 INFO [Transfer SnapMirror] Connecting to the cluster: DC3PNETAPP01/DestinationClusterIp
23:31:59.805 INFO [Transfer SnapMirror] Credentials successfully provided for 'DC3PNETAPP01.ad.xxxxxx.co.uk'
23:31:59.867 INFO [Transfer SnapMirror] Credentials successfully provided for 'DC3PNETAPP01.ad.xxxxxx.co.uk'
23:31:59.930 INFO [Transfer SnapMirror] Trying to connect to DC3PNETAPP01.ad.xxxxxx.co.uk using HTTPS on port 443 with timeout 60000
23:32:00.023 INFO [Transfer SnapMirror] Error while connecting HTTPSin NaServer::verify_server_certificate: server certificate verification failed: unable to get local issuer certificate
23:32:00.086 INFO [Transfer SnapMirror] Trying to connect to DC3PNETAPP01.ad.xxxxxx.co.uk using HTTP on port 80 with timeout 60000
23:32:01.195 INFO [Transfer SnapMirror] Error while connecting HTTPSin Zapi::invoke, cannot connect to socket
23:32:01.242 ERROR [Transfer SnapMirror] Command failed for Workflow 'Transfer SnapMirror relationship' with error :
Unable to connect to array

23:32:01.242 INFO [Transfer SnapMirror] ***** Workflow Execution Failed ***** 

If I run this test script, it works fine. I followed this forum post and configured the environmental variables. I can run the perl script from any directory, so they work fine.

I'm lost 😞 anyone got any ideas?

####

use NaServer;

#Set Array credentials below
$array="DC1PNETAPP01.ad.xxxxxxxx.co.uk";
$username = "xxxxxxxx";
$pass = "xxxxxxxx";

$server = NaServer->new($array, 1, 14);
$server->set_style('LOGIN');
$server->set_admin_user($username, $pass);
$server->set_transport_type('HTTPS');
$server->set_server_cert_verification(0);
$output = $server->invoke( "system-get-version" );
if (ref ($output) eq "NaElement" && $output->results_errno != 0) {
# failed to establish connection
print "Connection failed";

print $output->results_reason;
}
else {
# successfully connected
print "Connected successfully";
}

########

[amayacitta]

This bit makes me wonder about certificate trusts, how do I add them? they are trusted from the windows side, I've also added the roots into the Java certificate store. Do they need to go somewhere else as well? Maybe perl has a certificate root store too that doesnt use the windows one?

23:32:00.023 INFO [Transfer SnapMirror] Error while connecting HTTPSin NaServer::verify_server_certificate: server certificate verification failed: unable to get local issuer certificate

Perhaps this but I dont fully understand it..

http://docs.activestate.com/activeperl/5.26/perl/lib/LWP/Protocol/https.html

"If hostname verification is requested by LWP::UserAgent's ssl_opts, and neither SSL_ca_file nor SSL_ca_path is set, then SSL_ca_file is implied to be the one provided by Mozilla::CA. If the Mozilla::CA module isn't available SSL requests will fail. Either install this module, set up an alternative SSL_ca_file or disable hostname verification.

This module used to be bundled with the libwww-perl, but it was unbundled in v6.02 in order to be able to declare its dependencies properly for the CPAN tool-chain. Applications that need https support can just declare their dependency on LWP::Protocol::https and will no longer need to know what underlying modules to install."

[mbeattie]

Hi,

I tested the WFA certified PERL command "transfer snapmirror" on WFA 5.0.1 succuessfully in my lab. Is it possible that there is a certifcate issue on the destination vserver? Have you tried cloning the command and disabling hostname verification? EG

$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
my $wfa_util = WFAUtil->new();

Do you get the same error when updating the snapmirror relationship on a different vserver? Have you checked the certification expiration? EG

cluster2::> certificate show -fields expiration -type server -vserver vserver1_dr
  (security certificate show)
vserver     common-name serial           ca          type   subtype cert-name                    expiration
----------- ----------- ---------------- ----------- ------ ------- ---------------------------- ------------------------
vserver1_dr vserver1_dr 15B61346913114F7 vserver1_dr server -       vserver1_dr_15B61346913114F7 Wed Jul 29 13:58:34 2020

/Matt

[amayacitta]

Hi Matt,

I'm running 5.1.3212, the certificate hasn't expired and works fine on the WFA host outside of perl, I assume there is a way to import trusted root certificates and intermediates into the perl environment. Do you know how to do this? I've imported them into windows and Java  but this doesn't effect the perl runtime.

Happy to try disabling SSL verify once but I dont fancy doing that for all workflows, doing it once will at least help us understand if it is or isnt SSL that's the issue.

[amayacitta]

Seems this is how it's done, just trying to work out what is meant and if I can pernamently add a CA rather than doing it per workflow. If anyone has done this before let me know.

https://metacpan.org/pod/LWP::UserAgent#ssl_opts

[amayacitta]

OK so its 100% certificate based with Perl, if I set this to 0 then it works fine

$server->set_server_cert_verification(1);

I tried adding the root and subordinate certs to the PEM file for Mozilla as part of perl, which is in "C:\Program Files\NetApp\WFA\Perl64\lib\Mozilla\CA". This hasn't helped, I'm assuming there is another part to this to get the CA certs into the Perl runtime.

The WFA host itself has no issues connecting to the cluster management LIF cert, it's all fine. So it's 100% the Perl runtime and it's ability to accept the cert, what I dont understand is under normal conditions you would have self assigned certificates on the cluster managment LIF - I fail to see how this works out the box. Perhaps this can be taken up with engineering or a higher level? Is it possible something has changed in regards to certificates in the latest versions on WFA and Perl, I'm running the following version of Perl with 5.1.3212 Work Flow Automation.

PS C:\Windows\system32> perl -version

This is perl 5, version 26, subversion 3 (v5.26.3) built for MSWin32-x64-multi-thread
(with 2 registered patches, see perl -V for more detail)

Copyright 1987-2018, Larry Wall

Binary build 2603 [e95bda1d] NetApp provided by ActiveState http://www.ActiveState.com
Built May 29 2019 22:07:02

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

[paul_stejskal]

If WFA is failing and you've root caused the effect, then you can open a case. I don't work with the OnCommand products so a case is the best option.

[mjdalton1]

What do you get when you run this on your clusters?  Source and destination...

security certificate show -vserver * -common-name clustername*

Looks like a local certificate has expired.

[paul_stejskal]

Also what version of ONTAP?

[amayacitta]

The certs are all valid until 2029, they all work fine outside of the Perl runtime - via windows (on the WFA server) for example. This effects all 5 clusters, the versions are either 9.5P9 or 9.6P2.

The certs are issued by internal CA's which are trusted and chained OK. If there were issues on the NetApp end with the certificates it would effect more than just the Perl runtime.

DC3PNETAPP01::> security certificate show -vserver * -common-name DC3P*
Vserver    Serial Number   Certificate Name                       Type
---------- --------------- -------------------------------------- ------------
DC3PNETAPP01 
           1A0000000F7XXXXXXXXXX4000000000000F 
                           DC3NETAPP01                            server
    Certificate Authority: XXX-Sub02
          Expiration Date: Tue Dec 11 12:02:59 2029

I guess it's time to log a call, will sort Monday.

[mjdalton1]

What device is this then?

23:32:00.023 INFO [Transfer SnapMirror] Error while connecting HTTPSin NaServer::verify_server_certificate: server certificate verification failed: unable to get local issuer certificate

 I'd start there if you haven't already!

Happy Holiday's

[amayacitta]

Thats the DC3PNETAPP01 box I pasted in my last post, the one with the valid cert. Honestly, I think WFA configures Perl to accept default self-assigned certificates but not ones issued by a CA. I think the issue is the trust of the chain within the Perl environment. When I get it fixed, i'll post how 🙂

[amayacitta]

The NAServer relates to the perl module in this path:

C:\Program Files\NetApp\WFA\perl\nmsdk\NaServer.pm

There are loads of lines that relate to certificates, but I can't work out how to sort.

The dead giveaway is in the error is "unable to get local issuer certificate" - the issuer certificate is in the Perl module Mozilla::CA which is in the below path. I've imported the issuer certificate and root into there but I'm not convinced the NaServer module is calling it, according to the documetation it has to be called with SSLCACertificateFile. I'm giving up now... I'll log a call on Monday.

C:\Program Files\NetApp\WFA\Perl64\lib\Mozilla\CA\cacert.pem

Connection failedin NaServer::verify_server_certificate: server certificate verification failed: unable to get local issuer certificate

[amayacitta]

I've found a workaround whilst i wait for a response from NetApp support. The perl scripts as part of the NAServer Perl Module do not specifically call the Mozilla::CA roots pem file, which means with this version of WFA (havent tried others) if you issue your Cluster certificate from an internally trusted root and populate the file, it wont call the roots PEM file. Details on how to call the PEM file are here https://metacpan.org/pod/Mozilla::CA

However rather than re-coding the NetApp perl module, I've for now disabled HTTPS verification. I've done this by modifying line 1878 in the NaServer Perl module found under default path C:\Program Files\NetApp\WFA\perl\nmsdk.

It's been changed from this:

	$self->{enable_server_cert_verification} = $enable;
	$self->{enable_hostname_verification} = $enable;

To this

	$self->{enable_server_cert_verification} = 0;
	$self->{enable_hostname_verification} = $enable;

This won't be a great solution if your concerned about security and want a valid chain, however the data path will still remain encrypted via HTTPS. So it's good enough for now.

When I get to speak to the NetApp engineer, I'll see if they can create a fix to call the roots file as part of the module. I'd be really interested to know if this only effects WFA 5.1.3212.

[moep · Accepted solution]

I had a similar issue. This is new in WFA 5.1RC1, they activated the certificate check for perl workflows.

The root ca certificate store for perl resides in <installdir>\jboss\standalone\configuration\keystore\ontap.pem

Either put the filer certs or the root ca cert in the pem file.

[amayacitta]

Nice one, that works. Seems I got led down the wrong track with Mozilla::CA! Thanks for the real location of the referenced roots file.