With Insight in the rearview mirror, many of you who attended my sessions heard me talk about using nconnect with vSphere 8.0U2 and later achieving block-protocol performance levels with the simple NFS file protocols.
For those who haven’t heard of nconnect before (given how new it is in the VMware universe, it’s unsurprising that many VMware admins haven’t), let me give you a short primer.
Starting in ONTAP 9.8 (December 2020), NetApp added support for nconnect for NFS v3 and v4.X, which has been supported by Linux clients since kernel 5.3 (September 2019 and SUSE Linux the year prior). Nconnect allows an NFS client to establish multiple TCP connections from a single client interface (vmkernel port in this case) to a single server interface (ONTAP NFS LIF for example), allowing for distribution of I/O across the connections for improved performance.
When used with LACP ifgroups and TCP-port-based load balancing in ONTAP, and active-active uplinks with vmkernel ports in vSphere, NFS connections are balanced across multiple physical connections. Thereby allowing you to utilize your network resources to the maximum extent possible.
Compare the flow of IO with and without nconnect
Our internal testing has shown that you can achieve performance on par with block protocols using the same transport and number of active/optimized connections.
Refer to the solution guide for NFS nConnect feature with NetApp and VMware and NFS best practice and implementation guide | TR-4067 (netapp.com) as well as the previous blog post NetApp and VMware – Better than ever - NetApp Community. Broadcom also publishes Guidelines and Requirements for NFS Storage with ESXi (vmware.com). For more information from VMware by Broadcom, see knowledge base articles 91497 and 91479.
Full support was added for NFS v4.1 in vSphere 8.0 update 3.
Importantly nconnect can be utilized without downtime, reboots, or remounting of datastores. The number of connections can be scaled up or down as needed, from 1-4 (up to 8 with advanced settings).
Unfortunately, as of vSphere 8.0 Update 3, there is still no simple UI to manage your nconnect settings. To that end, for my own TME labs, I have written a basic PowerShell script that configures nconnect for you. It asks the user for the following items:
- vCenter to manage
- Credentials to use
- Which datacenter contains the hosts to update
- What the desired nconnect value should be.
The script configures the max allowed value to 8 then sets the value you entered for all datastore connects on all hosts in the indicated datacenter.
Note: ESXi has a hard-coded limit of 256 NFS connections, therefore consideration should be given to how many datastores you plan to use with nconnect as each additional connection counts towards the limit. That said, even increasing nconnect from 1 to 2 is well worth the effort because the increase in performance can be noticeable when you push the limits.
As a bonus, it also enables VAAI snapshot offload which eliminates the problem of vCenter-managed snapshots by using array-based disk clones instead of VMDK delta files. Note that VMs that have existing snapshots or that you don’t have permission to modify will throw an error in the script execution, but the script will just continue to the next VM.
Keep in mind that vCenter-managed snapshots that leverage VAAI being FlexCloned files are not the same as volume snapshots integrated with vCenter through the NetApp SnapCenter Plugin for VMware (SCV). SCV can also integrate tamper-proof snapshots (TPS) for extra ransomware protection and enable you to restore from secondary or tertiary snapshot copies on other systems. Both VAAI offloaded and SCV-integrated snapshots can be used to complement each other.
The only downside to VAAI snapshot offload is that vSphere doesn’t allow storage vMotion of VMs that have offloaded snapshots. You would have to delete the snapshots before you could storage vMotion that VM to another datastore. Feel free to remove that part if you don’t want to take advantage of snapshot offload.
And now for the script. This should help until the vCenter UI/API gap is addressed in the future. You may wish to modify it from the datacenter context to ESXi cluster context, depending on your use case.
#Load VMware PowerCLI core module if not already available
Import-Module VMware.VimAutomation.Core
# Prompt for the vCenter Server
$vCenter = Read-Host -Prompt "What vCenter do you want to connect to?"
Write-Output "You entered: $vCenter"
# Prompt for vCenter Server credentials
$cred = Get-Credential
# Connect to the vCenter Server using the provided credentials
Connect-VIServer -Server $vCenter -Credential $cred
# Prompt for the datacenter
$Datacenter = Read-Host -Prompt "Which datacenter do you want to update?"
Write-Output "You entered: $Datacenter"
# Prompt for nconnect value
$nconnect = Read-Host -Prompt "What do you want the nconnect value to be? (1-8)"
Write-Output "You entered: $nconnect"
Write-Output "Now updating VMs to enable snapshot offload"
#Enable VAAI snapshot offload
$vms = get-vm -Location $Datacenter
foreach ($vm in $vms) {
New-AdvancedSetting -Entity $vm -Name snapshot.alwaysAllowNative -Value TRUE -Confirm:$false -Force:$true
}
# Get all datastores and hosts in the datacenter
$V3Datastores = Get-Datastore | Where-Object {
$_.Type -eq "NFS" -and $_.FileSystemVersion -eq "3.0" -and $_.Datacenter -match ($Datacenter)
}
$V41Datastores = Get-Datastore | Where-Object {
$_.Type -eq "NFS" -and $_.FileSystemVersion -eq "4.1" -and $_.Datacenter -match ($Datacenter)
}
$vmhosts = Get-VMhost -Location $Datacenter
#Setting the maximum allowed nconnect value
Write-Output "Now updating the nconnect max value"
foreach ($vmhost in $vmhosts) {
Get-AdvancedSetting -Entity $vmhost -Name NFS.MaxConnectionsPerDatastore | Set-AdvancedSetting -Value 8 -Confirm:$false
}
# Loop through each datastore and set the nconnect value to $nconnect for both NFSv3 and NFSv41
Write-Output "Now updating NFS v3 datastores to $nconnect"
foreach ($vmhost in $vmhosts) {
foreach ($V3Datastore in $V3Datastores) {
$esxcli = Get-EsxCli -VMHost $vmHost -V2
# Set nconnect for NFSv3
$arguments1 = $esxcli.storage.nfs.param.set.CreateArgs()
$arguments1.volumename = $V3Datastore.Name
$arguments1.connections = $nconnect
$esxcli.storage.nfs.param.set.Invoke($arguments1)
}
}
Write-Output "Now updating NFS v4.1 datastores to $nconnect"
foreach ($vmhost in $vmhosts) {
foreach ($V41Datastore in $V41Datastores) {
$esxcli = Get-EsxCli -VMHost $vmHost -V2
# Set nconnect for NFSv41
$arguments2 = $esxcli.storage.nfs41.param.set.CreateArgs()
$arguments2.volumename = $V41Datastore.Name
$arguments2.connections = $nconnect
$esxcli.storage.nfs41.param.set.Invoke($arguments2)
}
}
# Disconnect from the vCenter Server
Disconnect-VIServer -Confirm:$false