- Overview
- NFS with Kerberos
- NFS over IPsec
- ONTAP enforcement of IPsec
- General IPsec storage information
- General IPsec client configuration
- IPsec with pre-shared key authentication
- IPsec with certificate authentication
- IPsec configuration notes for IPv6
- NFS over TLS
- NFS over TLS storage configuration
- NFS over TLS client configuration
Overview
NetApp is the most secure storage on the planet. With that in mind, let's look at the available technology for encrypting NFS traffic over-the-wire with NetApp ONTAP. The options include using NFS with Kerberos, running NFS over IPsec connections, and a nascent approach using NFS over TLS. While the setup for NFS with Kerberos is well documented, examples of end-to-end configuration for IPsec and NFS over TLS with ONTAP are a bit harder to find, so we will cover those details in this article.
If you want a quick summary of these options, here are the key points:
- NFS with Kerberos provides the most security features, including an option for over-the-wire encryption, but comes with the most complexity and the biggest hit to performance.
- NFS over IPsec is mature, fully supported, and comes in the middle concerning configuration complexity. NetApp already offers support for IPsec hardware offload cards (as of 9.16.1) in our new platforms to reduce the performance impact.
- NFS over TLS is the newcomer. The configuration complexity is the lowest, but it is not quite ready for production as of early 2025. The performance hit with IPsec and NFS over TLS is similar, and both provide better performance than NFS with krb5p.
If you are looking for details on how to configure over-the-wire encryption using these technologies with NetApp storage, read on!
NFS with Kerberos
We will start with a quick discussion of using NFS with Kerberos. This option has been available for many years and is well documented in TR-4616: NFS Kerberos in ONTAP. If you cannot trust who has root access on your NFS clients, then using NFS with Kerberos allows you to make sure that the users on your NFS client are who they say they are. NFS with Kerberos also provides for over-the-wire encryption, using the krb5p mount option. I won't go into the configuration steps for NFS with krb5p in this blog, since it is covered very well in TR-4616. While this is a great option for security, it also comes with the most complexity to configure and the largest performance hit of all the options for NFS over-the-wire encryption. In addition, there is no roadmap for improved performance of NFS with krb5p via hardware offload.
NFS over IPsec
Next up is running NFS over IPsec connections. ONTAP has supported IPsec for years, since the 9.8 release! In addition to pre-shared keys being supported since ONTAP 9.8, we have supported certificate authentication since 9.10.1. More recently, we support IPsec hardware offload using ONTAP 9.16.1 with our newer platforms. See the Evolution of the ONTAP IPsec Implementation for more details.
For the remainder of the IPsec section of this blog, I give some end-to-end examples of configuring IPsec with ONTAP and on the NFS client. There are multiple approaches to configuring IPsec, and many configuration options with the popular IPsec client software strongSwan. All traffic can be encrypted, or only certain ports. Policies can be configured between specific IP addresses, or between entire subnets. The approach I document in this article tries to keep the IPsec client configuration as generic as possible while allowing for simultaneous connections to multiple storage LIFs.
ONTAP enforcement of IPsec
When configuring NFS with Kerberos or using NFS over TLS, the NFS client can choose to mount an NFS export with the required options to enable these features or a standard mount can be performed with "sec=sys". This concept is different from IPsec in that once an IPsec security policy is defined and enabled for a LIF on the storage system, traffic is only allowed on that LIF via an IPsec connection.
General IPsec storage information
Below are some storage side commands that are useful to be aware of for any type of IPsec configuration:
# Enable ipsec on your cluster
security ipsec config modify -is-enabled true
# Check for active IPsec connections
security ipsec show-ikesa -node <nodename>
# Check for IPsec related errors
event log show -event ipsec.*
General IPsec client configuration
To use IPSec with ONTAP we need IPsec client software that supports IKEv2. In the Linux world, two popular software packages that fill this requirement are strongSwan and Libreswan. For my examples, I will be using strongSwan. To install strongSwan, run the following commands on a Red Hat compatible Linux distribution:
dnf install epel-release
dnf install strongswan
systemctl enable strongswan
systemctl start strongswan
Here are some notes about strongSwan that will apply to all of the IPsec configurations we show in this article:
- The strongSwan client configuration file path is /etc/strongswan/swanctl/swanctl.conf
- Debugging logging on the IPsec client can be seen with "grep charon /var/log/messages"
IPsec with pre-shared key authentication
Configuration with a pre-shared key is straightforward. In the following example, we configure a storage system policy for each of two LIFs on the SVM and use a subnet definition for the IPsec clients:
::> security ipsec policy create -vserver svm1 -name lif1_psk -local-ip-subnets 192.168.0.201/32 -remote-ip-subnets 192.168.0.0/24
{Enter your passphrase - I used netapp123netapp123netapp123}
::> security ipsec policy create -vserver svm1 -name lif2_psk -local-ip-subnets 192.168.0.202/32 -remote-ip-subnets 192.168.0.0/24
{Enter your passphrase - I used netapp123netapp123netapp123}
On the client, we use the following swanctl.conf configuration with a connection definition for each storage LIF and the pre-shared key:
connections {
svm1_lif1 {
children {
svm1_lif1 {
esp_proposals = aes256gcm16
mode = transport
start_action = trap
# Connection from local subnet to lif1.
local_ts = 192.168.0.0/24
remote_ts = 192.168.0.201/32
}
}
# Connection from local subnet to lif2.
local_addrs = 192.168.0.0/24
remote_addrs = 192.168.0.201/32
proposals = aes256-sha384-ecp384
local {
auth = psk
}
remote {
auth = psk
}
}
svm1_lif2 {
children {
svm1_lif2 {
esp_proposals = aes256gcm16
mode = transport
start_action = trap
# Connection from local subnet to lif2.
local_ts = 192.168.0.0/24
remote_ts = 192.168.0.202/32
}
}
# Connection from local subnet to lif2.
local_addrs = 192.168.0.0/24
remote_addrs = 192.168.0.202/32
proposals = aes256-sha384-ecp384
local {
auth = psk
}
remote {
auth = psk
}
}
}
secrets {
ike-svm1 {
# This must match the pre-shared key entered for storage policy.
secret = netapp123netapp123netapp123
# The ike id must match the storage policy remote-ip-subnets.
id = 192.168.0.0/24
}
}
Now run "swanctl --load-all" and then "swanctl --list-conns" to make sure your connections are properly loaded. If all is well, you should be able to successfully ping the storage LIFs from the IPsec client,
IPsec with certificate authentication
Configuration with certificate authentication is a bit more involved due to the requirement for creating and configuring certificates both on the storage LIFs and the IPsec clients. If you are familiar with PKI, you can create and manage the certificates however you wish. In this example I use the certificate management tools built into ONTAP. Note that when copying a public key certificate or a private key, always include the "-----BEGIN ..." and "-----END ..." lines. First, we will create a certificate authority (CA), add that CA to the local ONTAP ipsec configuration, and copy the CA public key certificate to the IPsec client strongSwan configuration directory:
security certificate create -vserver svm1 -type root-ca -common-name svm1 -cert-name svm1_ipsec_ca
security ipsec ca-certificate add -vserver svm1 -ca-certs svm1_ipsec_ca
security certificate show -vserver svm1 -cert-name svm1_ipsec_ca -instance
{Copy the public key certificate to a file on the IPsec client at /etc/strongswan/swanctl/x509ca/svm1_ipsec_ca.pem}
Next, we will create the certificate to use for our storage LIF. Repeat the following steps for each storage LIF that is part of your IPsec policy, making sure to use the proper FQDN and IP address when generating the certificate signing request:
security certificate generate-csr -common-name svm1_lif1.demo.netapp.com -algorithm RSA -hash-function SHA256 -ipaddr 192.168.0.201 -dns-name svm1_lif1.demo.netapp.com
{Save each certificate request and private key}
security certificate sign -vserver svm1 -ca svm1 -ca-serial 1821A2A76A371FFC -expire-days 360
{Enter each certificate to sign and get signed certificate}
security certificate install -type server -vserver svm1
{Enter each signed certificate and private key}
We also need a certificate to use for the Linux IPsec client. We will again use the ONTAP certificate management functionality to create the certificate, and then place the public key certificate and private key in the correct location for strongSwan on the IPsec client:
security certificate generate-csr -common-name linux1.demo.netapp.com -algorithm RSA -hash-function SHA256 -ipaddr 192.168.0.61
{Save certificate request and private key}
security certificate sign -vserver svm1 -ca svm1 -ca-serial 1821A2A76A371FFC -expire-days 360
{Enter certificate to sign and get signed certificate}
{copy the signed public key certificate to /etc/strongswan/swanctl/x509/linux1.demo.netapp.com.pem}
{copy the private key to /etc/strongswan/swanctl/private/linux1.demo.netapp.com.key}
OK! With all the certificates in place, we use the following swantctl.conf on the IPsec client:
connections {
svm1_lif1 {
children {
svm1_lif1 {
esp_proposals = aes256gcm16
mode = transport
start_action = trap
local_ts = 192.168.0.0/24
remote_ts = 192.168.0.201/32
}
}
local_addrs = 192.168.0.0/24
remote_addrs = 192.168.0.201/32
proposals = aes256-sha384-ecp384
local {
auth = pubkey
id = "CN=linux1.demo.netapp.com"
certs = linux1.demo.netapp.com.pem
}
remote {
auth = pubkey
id = "CN=svm1_lif1.demo.netapp.com"
}
}
svm1_lif2 {
children {
svm1_lif2 {
esp_proposals = aes256gcm16
mode = transport
start_action = trap
local_ts = 192.168.0.0/24
remote_ts = 192.168.0.202/32
}
}
local_addrs = 192.168.0.0/24
remote_addrs = 192.168.0.202/32
proposals = aes256-sha384-ecp384
local {
auth = pubkey
id = "CN=linux1.demo.netapp.com"
certs = linux1.demo.netapp.com.pem
}
remote {
auth = pubkey
id = "CN=svm1_lif2.demo.netapp.com"
}
}
}
Now run "swanctl --load-all" and then "swanctl --list-conns" to make sure your connections are properly loaded. If all is well, you should be able to successfully ping the storage LIFs from the IPsec client.
IPsec configuration notes for IPv6
I've also tested IPsec with IPv6, and ran into a couple of strongSwan configuration requirements to make this work. The first issue is that we need to bypass IPv6 NDP NS and NA traffic in the swanctl.conf as shown in this snippet of the configuration file:
{Add this at the start of the connections block}
# Bypass IPv6 NDP NS and NA traffic.
# As per https://docs.strongswan.org/docs/5.9/config/IPv6Ndp.html
ndp {
children {
ns {
mode = pass
start_action = trap
local_ts = ::/0[ipv6-icmp/135]
remote_ts = ::/0[ipv6-icmp/135]
}
na {
mode = pass
start_action = trap
local_ts = ::/0[ipv6-icmp/136]
remote_ts = ::/0[ipv6-icmp/136]
}
}
}
One more change that I found from my testing with IPv6 is that simple IP address endpoints (without subnet notation) must be given in the local_addrs and remote_addrs files of swanctl.conf, as shown in this snippet:
local_addrs = fd12:1234:5678:1::2
remote_addrs = fd12:1234:5678:1::4
NFS over TLS
Finally, we will showcase the use of NFS over TLS. The NFS over TLS feature was introduced in ONTAP 9.15.1 as a Tech Preview. This means that it is not yet ready for production, and full GA support for this feature in ONTAP will come in a future release. You should also be aware that this is a very new feature on the Linux NFS client side as well, as it first shows up in RHEL as a Technology Preview in RHEL 9.4.
NFS over TLS storage configuration
Let's look at the configuration steps to get NFS over TLS running. NFS over TLS relies on certificates, and we will need to configure a unique certificate for each LIF in the SVM that will handle NFS over TLS traffic. If you are familiar with PKI, you can create and manage the certificates however you wish. In this example, I will simplify the setup by using the certificate management tools built into ONTAP. First, we will create a certificate authority (CA) in the SVM and display the public key certificate of the CA to be used later on our NFS client. Note that when copying a public key certificate or a private key, always include the "-----BEGIN ..." and "-----END ..." lines.
security certificate create -vserver svm1 -type root-ca -common-name svm1_ca
security certificate show -vserver svm1 -common-name svm1_ca -type root-ca -instance
Next, we will create a certificate for one of our SVM LIFs and configure the LIF to use this certificate for NFS over TLS:
security certificate generate-csr -common-name svm1_lif1.demo.netapp.com -algorithm RSA -hash-function SHA256 -ipaddr 192.168.0.61 -dns-name svm1_lif1.demo.netapp.com,svm1_lif1,svm1
{Save certificate request and private key}
security certificate sign -ca svm1_ca -vserver svm1 -ca-serial 181F8F545154B6BA -expire-days 364
{Enter certificate to sign and get signed certificate}
security certificate install -vserver svm1 -type server -cert-name svm1_lif1.demo.netapp.com
{Enter signed certificate and private key}
nfs tls interface enable -vserver svm1 -lif svm1_lif1 -certificate-name svm1_lif1.demo.netapp.com
The above commands will be required for each LIF on which you want to run NFS over TLS traffic. Also, a few notes are in order about the security certificate generate-csr command above:
- The -common-name argument must be the FQDN of the LIF IP address.
- You must use the correct LIF IP address in the -ipaddr argument.
- You should configure the -dns-name argument to include any hostnames that may be used in NFS mounts from the client. This includes configurations where you may use multi-homed DNS entries so that multiple LIF IP addresses are returned for a single hostname.
NFS over TLS client configuration
Now, on the Linux NFS client, we have to first install the ktls package. In my example, I'm using a Red Hat 9.4 compatible client:
dnf install ktls-utils
systemctl enable tlshd
systemctl start tlshd
Next, remember that CA public key certificate I told you to make a copy of from the output of the "security certificate show -vserver svm1 -common-name svm1_ca -type root-ca -instance" command? We need to place it on the Linux client and run a command as follows:
vi /etc/pki/ca-trust/source/anchors/svm1_ca.pem
{Paste the CA public certificate that was used to sign LIF certificates on storage side}
update-ca-trust extract
Now we should be able to mount an NFS export using TLS! Here is the syntax to mount with NFSv3 over TLS:
mount -o nfsvers=3,xprtsec=tls svm1_lif1:/vol1 /vol1
That's it! If you are using IPv6, the procedure for NFS over TLS works the same way - substitute IPv6 IP addresses as required when creating your certificates.
I hope you found all of these details useful! Hit me in the comments if you have any questions on these technologies or how the configuration works.