Now that Windows 2012 has been officially released, we are in the process of releasing NetApp host side product stacks supporting Windows 2012 (WS2012).
This includes a newer version of SnapDrive for Windows (SDW), SnapManager for Hyper-V (SMHV), On Command Plug-In for Microsoft (OCPM), SMI-S Agent.
I will be writing a series of these blog posts(every month) covering some of the unique value add, NetApp Windows host side product(s) is bringing to our customers running on Windows Server 2012
One of the key features Microsoft released in Windows Server 2012 is the CSVFS (Cluster Shared Volume File System). As you all know the CSV was introduced in Windows 2008 R2. Just to recap, CSV 1.0 supports following functionalities. CSV presents a consistent file namespace to all cluster nodes and allows multiple cluster nodes to concurrently access a LUN(Logical Unit Number) on a shared storage system in Microsoft’s Failover Cluster. One of the primary design goals was to support flexible live migration of virtual machines in a Windows Failover Cluster, by providing the facility to host multiple Virtual Machines (VMs) on Clustered Shared Volumes (CSV)
Some of the key concepts about CSV are given below, these terminologies has direct implication on the way the backup is performed on WS2012.
CSV has the concept of co-ordinator node, Co-ordinator node typically is the node which owns cluster “Physical Disk” resource of a CSV volume. It is the one node that is synchronizing all access to that volume from all nodes across the cluster. The VSS infrastructure is present on the coordination node, and it is the only source node in the cluster that is safe to conduct backups from.
All other nodes in the cluster that are not the owner of the Physical Disk resource conduct data access through a technique that is referred to as “Direct I/O”.
Direct I/O is the process used by CSV to perform I/O operations from a VM to the storage device with enhanced performance. This is accomplished by splitting the transaction and sending NTFS metadata operations to the coordination node over the network while sending data I/O directly to the disk class driver for the storage device.
By sending data directly from a file system mini-filter driver called CSVFilter.sys to the disk class driver, CSV bypasses all local file system and volume/partition processing on non-coordination nodes. This includes the VSS infrastructure, resulting in CSV and Direct I/O being incompatible with the environment required by the VSS infrastructure. In Windows 2008 R2, a backup cannot be initiated by a Requester on a non-coordination node.
During the CSV VSS backup, Re-directed I/O will be enabled so that all I/O is sent over the network and synchronized by the coordination node. In this scenario both the data and metadata from all the partner cluster nodes are send to the coordinator node over the SMB client redirector present on each cluster node. As soon as the backup is complete the Re-directed I/O is stopped and the directed I/O resumes. Re-directed I/O is also used to send the data/metadata to the underlying storage through the coordinator node, if a cluster node loses direct connection to the storage.
CSV Hyper-V backup will create application consistent backups on the each VM owner node. CSV owner ship is moved to the VM owner node as a part of the backup process. Hyper-V VSS Writer will then co-ordinate the Freeze and Thaw operations in the Hyper-V guest, and subsequent hardware Snapshot is taken from the Hyper-V parent using the DataONTAP VSS hardware provider. This result in a hardware snapshot created for each Windows cluster node.
This has introduced several scalability and space efficiency issues when the number of nodes in the cluster is increased. Assuming that the same solution exists in WS 2012, the backup solutions will not really scale at all. Hyper-V VSS backup needs to create one hardware snapshot per VM owner node which would have resulted in 64 hardware snapshot for one cluster wide backup. The following diagram is the current state of affairs in WS2008 R2 for an application consistent backup. As you can see there are two nodes involved in the VSS Snapshot set.
In the example below there are two Hyper-V hosts Host1, Host2. VM1, VM2 running on Host1 and VM3, VM4 running on Host2. Application consistent backup needs to be performed on the VM owner node. The requestor prepares the backup on each node in the cluster in a serial fashion. The VSS requestor calls a "special CSV API" named ClusterPrepareSharedVolumeForBackup, this API moves the CSV to the backup node and CSV will go into the "Re-directed I/O" mode until the time of the backup. Because of this reason a parallel backup from another node in the cluster is not allowed.
If another node attempts a backup at the same time an error is thrown to the VSS requestors. Although CSV errors out there is no in build locking mechanism in CSV which prevents any other node to abort the backup which has been started by a different node. Because of this reason the requestors are required to code extra caution in the code to preserve the locking themselves.
Assume the CSV failover to the Host1 and the backup is started and the CSV goes into the redirected mode. The Hyper-V VSS writer on the Host1 will prepare the VM for the backup. Hyper-V writer then contacts the VSS Integration component inside the VM guest and will prepare a software shadow copy of applications inside the guest. The hardware provider will then be contacted from Hyper-V parent to create the actual hardware snapshot of the guest VHD volume.
Once this process is done, the "auto recovery" process begins, the hardware provider will be contacted by the VSS infrastructure to mount the CSV snapshot LUN from the back upset to prepare for auto recovery. Once the auto recovery is completed the VM VHD will be in a state of the application consistent and then the hardware provider will be contacted to make the final auto recovered changes read only.
This is the laborious backup process of Hyper-V VMs running on CSV volumes. The same process continues for Host2 and the backup is then complete from the VSS requestor's perspective. This makes the Hyper-V backup complex for CSV. Performance is big issue since multiple VSS initializations/backup complete are done on each node in the cluster. This type of solution is hard to scale as the number of nodes and VMs in a cluster increases like in WS2012.
See the sections below to see how MS has re-architected Cluster Shared Volume to address these limitations.
CSV 2.0 introduced a new full blown File System CSVFS. CSV 2.0 in Windows 2012 has the following features comparing to its predecessor. Infact MS claims CSV 2.0 is completely rewritten from the scratch. Following are the high lights of the CSV 2.0 file system.
CSV 2.0 management in WS 2012 is much simplified comparing to CSV in Windows 2008 R2.
CSV 2.0 has introduced a ground breaking feature named "Distributed application consistent backup" in a Windows Failover cluster. This allows the customers to backup all the VMs in a cluster to be consistent in one application consistent backup. To support this feature of course there needs to be a co-ordination mechanism in Windows to co-ordinate the backup across multiple cluster nodes.
Microsoft has introduced two new components in the VSS framework supporting the CSV 2.0 application consistent backup. A new CSV Writer and CSV Provider have been added into the WS 2012 to achieve the distributed backup. CSV Writer serves the Component level metadata from non-requesting node for CSV volumes and act as a proxy by including the Hyper-V writers from remote node for the backup session. Also the metadata of the Virtual machines reported by the CSV writer defers comparing to the Hyper-V writer. This puts extra work on the requestors to understand the component schema and do the necessary backup preparation mechanisms.
CSV Provider co-ordinates the VSS Freeze and Thaw from all the Hyper-V writers on the partner cluster nodes to make the VM in an application consistent state. Also the CSV provider makes sure that CSV shadow copy volume is writable for the partner node Hyper-V writers during the auto recovery process explained earlier.
Following image shows the newly added CSV provider in WS2012 system which helps to co-ordinate the distributed backup.
The following block diagram shows the SnapManager for Hyper-V backup sequence including the CSV 2.0 Writer, and Provider to create an application consistent backup of multiple VMs across the Windows Failover Cluster. In this scenario, the CSV writer queries all the partner node VM metadata for the VSS requestor on the backup node. The VSS hardware provider is only contacted on the backup node and finally only one hardware snapshot is created for the entire CSV volume.
This feature was demonstrated at Microsoft TechEd Orlando Florida, You can view the complete session in the following link.
Cluster Shared Volumes Reborn in Windows Server 2012: Deep Dive
I will be talking about other great features implemented in SnapManagers/OnCommand Plug-In for Microsoft in the upcoming blog posts in about a month.