Introduction to FPolicy
Worldwide unstructured data in growing at an exponentially rate. IDC predicts that by 2020 amount of digital data can be a mammoth 40 Zettabytes(ZB). Significant percentage of this will be file based unstructured data, which is in fact one of the fastest growing segment of digital data. Unstructured file data is expected to grow at CAGR 61.5% YoY posing significant challenges to Organizations and regulators. Managing this data requires adhering to best practices in the industry and leveraging the expertise in the ecosystem. Clustered Data ONTAP 8.2 enables this through its File Policy framework FPolicy.
FPolicy is a file access notification framework which allows file screening of NFS and CIFS accesses. Introduced in clustered Data ONTAP 8.2 for NetApp scale out architecture this enables a rich set of use-cases working with our selected partners. FPolicy requires that all the nodes, in the cluster, are running Data ONTAP 8.2 or later. FPolicy supports all the SMB versions like SMB1.0 (aka CIFS), SMB 2.0, SMB 2.1 and SMB 3.0 and major NFS versions like NFS V3 and NFS V4.0.
FPolicy natively supports simple file blocking use case which enables administrators to restrict end users to store unwanted files. Example: An admin can block audio and video files from getting stored in Data Centers and thus saving precious storage resources. This feature blocks files only based on extension and for more advanced features partner solutions have to be considered. The framework generates external on-wire notifications which can either alter the file access path or just report the file access: enabling a diverse set of use-cases. Partner solutions, built around the framework, provides rich set of use-cases like data governance, File access reporting, User and directory quota, File tiering, File Replication, encryption and many more.
The notification framework is platform agnostic and the partner application can run on Windows, UNIX, Linux or Mac platform. The underlying operating system can be either a physical server or a virtual one. FPolicy best practices need to be followed while developing and deploying the solution. FPolicy best practices are part of CIFS best practices Technical Report
Kindly be aware that the “Notification Schema” and “Sample Code” provided as part FPolicy SDK are legally protected resources. Currently they are available only to NetApp Technology Partners. Generic FPolicy configuration ONTAPIs are covered in NMSDK 8.2 documentation and available to all NetApp customers and partners.
The first step to implementing an FPolicy use-case is to configure a resident FPolicy policy on the cluster. This policy can be configured either in cluster scope or Storage Virtual Machine (SVM) scope. When configured in cluster scope they act as template and can be applied to all the SVMs in the cluster. Resident policies are configured either by Command Line Interface (CLI) or by FPolicy APIs, which are documented in NMSDK documentation for clustered Data ONTAP 8.2 clustered mode.
Resident policy on the controller is divided into four containers based on functionality. They are
External Engine: This container manages external communication with FPolicy server application.
Events: This container captures information about protocols and file operations monitored for the policy
Policy: Main container which associates different constituents of the policy and provides platform for policy management like policy enabling and disabling
Scope: This container define the storage objects on which the policy will act. Ex: Volumes, sharers, exports and file extensions
Workflow for creating resident policy is as shown in the diagram. You need to create external engine or Events before creating a policy. Only when policy is defined you can associate a scope to it.
Once the scope is created policy needs to be enabled with the sequence number. The sequence number helps to define the priority of the policy in a multi-policy environment. With sequence of 1 having highest priority and 10 having the lowest.
Native File Blocking
FPolicy can be used for native file blocking. Native blocking policy can be configure in a few simple steps using command line interface (CLI). Configuring and managing these policies will be supported in future version of System Manager as well. They can be configured either at cluster context or SVM context based on the storage requirement. Let us look at configuring policy in SVM context
Configuring Policy Engine
External engine defines the IP address and port of the FPolicy server. It defines the characteristics of notification channel like
- sync or async mode of communication
- SSL or plain text communication
Native blocking use case doesn't have any dependency on external engine and instead uses a preconfigured template 'native' in place of external engine
Configuring Policy Events
we recommend monitoring these file operations for the native file blocking use case.
File Operations to Monitor
Folder operations to Monitor
Create, Open, Rename
Create, Write, Rename, Symlink
Create, Open, Rename, Symlink
This command will create a event for monitoring CIFS protocol. You can create containers for NFSV3 and NFSV4 and associate with a single policy
clus2: fpolicy policy event create -vserver <<vserver-name> -event-name <<event-name>> -protocol cifs -file-operations create,open,rename -volume-operation false
Configuring Policy Container
This is the placeholder that will associate Event and External Engine with Scope. This has other flags that can be set to define the policy behavior like is-mandatory, allow-privileged-access and others. For Native Blocking use case these fields are irrelevant. In Native blocking use case sequence number is irrelevant and highest priority is assigned to the policy.
This command will create the policy event with preexisting external engine template native.
clus2::> fpolicy policy create -vserver <<vserver-name>> -policy-name <<policy-name>> -events <<event-name>> -engine native
Configuring Policy Scope
Scope will define the storage objects on which the policy will be active. The storage objects can be volumes, CIFS shares or NFS exports. One can choose the file extension that will be monitored by the policy. If you like to monitor all the files you can either live it blank or give '*' . File extension can be a regular expression and support '*' and '?' as special characters. Scope of policy on storage objects can be controlled with the include and exclude lists. Between two lists exclude list has higher priority over include list.
Some operating systems like Mac has folders with extension. Monitoring folders with extension can be controlled with is-file-extension-check-on-directories-enabled option available in advanced mode. Default value of the option is false and operations on folders will be monitored by default. The default response is to block access; share access will be blocked. To overcome this limitation, while configuring native blocking, we should enforce extension on directories as well by setting the option to true
This command will define the policy scope that monitors only file operation on text and MS Doc files. Scope is defined on all shares except share1
Note: Based on the terminal user may not be able to enter special characters in such cases you may have to press escape key before keying in the characters
clus2::*> fpolicy policy scope create -vserver <<vserver-name>> -policy-name <<policy-name>> -shares-to-include "*" -shares-to-exclude share1 -file-extensions-to-include "txt,doc*" -is-file-extension-check-on-directories-enabled true
Policy can be enabled with a simple command using the available sequence number. Sequence number is irrelevant for native blocking use case but an arbitrary value from 1-10 has to be provided. Once enabled policy should be configured and running.
clus2::> fpolicy enable -policy-name <<policy-name>> -sequence-number <<sequence number>>
Enhancements to 7-Mode
Significant enhancements are made to FPolicy implementation in clustered Data ONTAP compared to earlier implementation in Data ONTAP 7-Mode. They can be broadly classified on these lines
- No-dependency on resources like pblks
- File notification are sent in XML over TCP format, making it both platform independent and less taxing on network
- Ability to clearly set policy priority in multiuse-case scenario
- Ability to monitor directory extension. Ex: Mac environment
- Ability to apply back pressures to reduce load on Servers
- Ability to filter the notification on SVM side that enables optimal utilization of resources and better overall solution throughput.
- Securing the notification channel through SSL. Notification channel can be plain text, dual side or single side encryption
- Enhanced FPolicy safeguards to manage client outages in case of Network or server disruption
For some of the advanced use-cases we integrate with our partners to provide an end to end solution and incorporate our best practices. Partner solution leverage FPolicy framework to offer additional use cases on top of the native use cases provided by Data ONTAP. The solutions can be in the space of security, storage management, access reporting, archiving, File Replication and others. Some of our active partners in this space
- Varonis Data Governance and Data Analytics solution DatAdvantage
- NTP Software QFS for Quota and File Screening
- Many more to come…