Putting the “S” in SMT: NetApp, Cisco, and VMware Enhance Security for Multi-Tenant Environments

by on ‎2010-11-17 11:57 AM

Whether you're deploying large-scale virtualization, private cloud, or public cloud, security can be a challenge. Cisco, NetApp, and VMware recognized security as a critical issue for shared infrastructure and have been working together to address the problem for several years.

You might have read previous Tech OnTap® articles on our joint secure multi-tenancy (SMT) solution. In January 2010, Chris Naddeo of Cisco wrote extensively about its quality of service (QOS) capabilities. An article a few months later talked about deploying our first-generation SMT solution in cloud environments according to the detailed Cisco® Validated Design (CVD). However, neither of these articles provided full details on the security capabilities of SMT.

Now, less than a year since the initial release, we've created an improved version of SMT that increases deployment flexibility and further extends the capabilities and security of the design. In this article we'll discuss the security architecture of our enhanced secure multi-tenancy design. The enhanced SMT design consists of a number of foundational components plus optional components that you can use to flexibly tailor your environment for specific security requirements. The details of these components are explained later in the article.

Understanding SMT Security

The initial SMT design focused on four pillars:

  • Availability
  • Secure separation
  • Service assurance
  • Service management

Our enhanced SMT architecture continues this focus. While we've made significant advances in all areas since the first release, we want to talk specifically about secure separation for multi-tenant environments. For our purposes, this means environments where infrastructure is shared by tenants—which could be defined as separate customers, different business units, departments, applications, or any other entities that need to be securely isolated.

Figure 1)
Services provided by the secure separation component of enhanced SMT (click to expand).

Secure separation makes sure that one tenant does not have access to another tenant's resources, such as virtual machines (VMs), networks, and storage. Each tenant must be securely separated from every other tenant.

The enhanced SMT architecture prevents one tenant from having access to another's data and also prevents a tenant from having access to the administrative features of the shared infrastructure. The key innovation that our SMT—and now second-generation SMT—implementation brings to secure separation is the ability to provide complete separation at every layer: compute, network, and storage. Other solutions are able to provide only pockets of isolation within the infrastructure rather than end-to-end tenant separation.

The main security principles that are implemented in this architecture include:

  • Isolation: Isolation provides the foundation for security for the multi-tenant data center. Depending on the goals of the design, it can be achieved through the use of firewalls, access lists, VLANs, virtualization, storage, and physical separation. A combination of these can provide the appropriate level of security enforcement to the applications and services within different tenants.
  • Policy enforcement and access control: Within a multi-tenant environment, the issue of access control and policy enforcement requires careful consideration. Capabilities of devices and appliances within each layer of the architecture can be leveraged to create complex policies and levels of access control that can enhance secure separation within each tenant.
  • Visibility: Shared infrastructures are becoming very fluid in the way they scale to accommodate new virtual machines and services. Server virtualization and technologies, such as VMotion™, allow new servers to be deployed and to move from one physical location to another with little manual intervention. When these machines move and traffic patterns change, it can create a challenge for security administrators to actively monitor threats. This architecture leverages the threat detection and mitigation capabilities that are available at each layer of the network to gather alarm, data, and event information and dynamically analyze and correlate the information to identify the source of threats, visualize the attack paths, and suggest and optionally enforce response actions.
  • Resiliency: Resiliency implies that endpoints, infrastructure, and applications within the multi-tenant environment are protected and can withstand attacks that can cause service disruption, data disclosure, and unauthorized access. Proper infrastructure hardening, providing application redundancy, and implementing firewalls are some of the steps needed in order to achieve the desired level of resiliency.

The network, compute, storage, and management components within this architecture provide features and capabilities that together form the framework that makes sure of secure separation. Figure 2 illustrates the security architecture used in this design, highlighting the functional areas of the solution, its components, and their corresponding security features.

Figure 2)
Enhanced SMT architectural framework (click to expand).

Enhanced SMT addresses the different traffic patterns that exist within the shared environment. The traffic flows are logically divided into two distinct categories: east-west and north-south. In the typical case, VMware® vShield provides security for east-west traffic, while some combination of Cisco security services is used to provide the necessary level of security for north-south traffic. These components are described in more detail later.

Figure 3)
Security for "east-west" versus "north-south" traffic (click to expand).

North-south traffic flows are either ingress or egress in relation to the SMT infrastructure and are commonly client-to-server in nature. This traffic traverses the data center and is readily exposed to any number of services in its path, including firewalling, load balancing, intrusion detection, and network analysis devices. In a multi-tenant environment, traffic between tenants may also be forced through the shared infrastructure network services. This functionality is ideal as each tenant policy is uniformly applied between every tenant. The exposure of ingress-egress traffic flows to security services in the data center is dependent upon application specific requirements and the overall security policies of the enterprise.

It is important to note the flexible nature of this architecture, where a security architect can use any combination of security features as described in the later section on enhanced SMT security options to create security offerings. We refer to this flexibility as defense in depth.

East-west traffic refers to the communication between servers within the data center. Securing interserver communication can be an application-based requirement or an enterprise-based requirement. Typically, enterprise-class applications require more availability, scalability, and/or processing power than a single server instance can provide. To address these issues, application developers use dedicated server roles. Each role is specialized and dependent on other servers. Virtual machines fully support this application model. In SMT, server-to-server flows between virtual machines may occur within a single tenant container or between tenants.

To optimize east-west traffic patterns within a virtualized data center, it is recommended to use a virtual firewall appliance such as vShield to provide secure connectivity between virtual machines. This service may securely support intra- or intertenant communication. For example, a virtual firewall can provide secure connectivity for tenant virtual machines that need access to infrastructure services such as Active Directory residing in a common infrastructure tenant.

Enhanced SMT Foundation Components

The basic infrastructure used for SMT consists of:

  • Cisco Nexus® data center switches
  • Cisco Unified Computing System™ (Cisco UCS)
  • Cisco Nexus 1000V Distributed Virtual Switch
  • NetApp® storage
  • VMware vSphere™
  • VMware vCenter™ Server

The infrastructure needed for SMT is essentially identical to the recently announced FlexPod™ architecture, which is described in a companion article in this issue of Tech OnTap. FlexPod is a great starting point for enhanced SMT, but doesn't provide SMT by default.

The enhanced SMT foundation requires several additional components on top of the basic infrastructure components.

The VMware vShield family of security solutions provides virtualization-aware protection for virtual data centers and cloud environments, strengthening application and data security, improving visibility and control, and accelerating IT compliance efforts. The vShield family includes:

  • vShield Edge, which provides network edge security and services to isolate the virtual machines in a port group from the external network. The vShield Edge connects isolated tenant stub networks to shared (uplink) networks and provides common perimeter security services such as DHCP, VPN, and NAT.
  • vShield App provides firewalling between virtual machines by placing a firewall filter on every virtual network adapter. The firewall filter operates transparently and does not require network changes or modification of IP addresses to create security zones. Rules can be written using vCenter groupings such as data center, cluster, resource pools, and vApps or network objects such as port group and VLAN to reduce the number of firewall rules and make the rules easier to track.
  • VMware vShield Zones is a centrally managed, stateful, distributed virtual firewall bundled with vSphere 4.1 that takes advantage of ESXi host proximity and virtual network visibility to create security zones. By leveraging various VMware logical containers, it is possible to greatly reduce the number of rules required to secure a multi-tenant environment and therefore reduce the operational burden that accompanies the isolation and segmentation of tenants and applications. This new way of creating security policies closely ties to the VMware virtual machine objects and therefore follows the VMs during VMotion.
  • In addition to being an endpoint and asset-aware firewall, the vShield Zones contain microflow-level virtual network reporting that is critical to understanding and monitoring the virtual traffic flows and implement zoning policies based on rich information available to security and network administrators.

  • The flow monitoring feature displays allowed and blocked network flows at application protocol granularity. This can be used to audit network traffic and as an operational troubleshooting tool.

Cisco Nexus 1000V. While the Cisco Nexus 1000V switch is a standard FlexPod component, we are discussing it explicitly here because of its security capabilities. The Cisco Nexus 1000V is a software switch on a server that delivers Cisco VN-Link services to virtual machines hosted on that server. It takes advantage of the VMware vSphere framework to offer tight integration between server and network environments and help make sure of consistent, policy-based network capabilities to all servers in the data center. It allows policy to move with a virtual machine during live migration, making sure of persistent network, security, and storage compliance, resulting in improved business continuance, performance management, and security compliance.

As data centers become more fluid in nature, it becomes more difficult for security administrators to actively monitor threats within the infrastructure. To address this the Cisco Nexus 1000V virtual distributed switch supports monitoring using NetFlow, Switch Port Analyzer (SPAN), and Encapsulated Remote SPAN (ERSPAN). These features enable the Cisco Nexus 1000V to export VM traffic flow data from the virtual environment to traditional external analysis devices or advanced virtual service blades. In either case, the Cisco Nexus 1000v offers improved visibility for security teams to identify the source of threats, visualize attack paths, and ultimately act.

The Cisco Nexus 1010 Virtual Services Appliance hosts the Cisco Nexus 1000V Virtual Supervisor Module (VSM) and supports the Cisco Nexus 1000V Network Analysis Module (NAM) Virtual Service Blade to provide a comprehensive solution for virtual access switching. The Cisco Nexus 1010 provides dedicated hardware for the VSM, making virtual access switch deployment easier while enhancing performance and operational efficiency.

NetApp MultiStore® allows you to create separate and completely private logical partitions on a single NetApp storage system in discrete administrative domains called vFiler™ units. These vFiler units have the effect of making a single physical storage controller appear to be many logical controllers. Each vFiler unit can be individually managed with different sets of performance and policy characteristics. You can leverage MultiStore to enable multiple customers to share the same storage resources with minimal compromise in privacy or security. Administrative control of the virtual storage container can even be delegated directly to the customer. MultiStore was the subject of a recent Tech OnTap article.

Enhanced SMT Security Options

In addition to the standard components described in the previous section, you can also choose from a variety of security-related optional components to satisfy particular security requirements.

VMware vCloud Director (VCD) builds upon the VMware vSphere foundation and exposes virtualized shared infrastructure as multi-tenant virtual data centers that are completely decoupled from the underlying hardware. What this means is that using this technology, you can deliver standardized IT services on shared infrastructure through a Web-based catalog. vCloud Director was the subject of another recent Tech OnTap article.

The Cisco Catalyst® 6500 Series Virtual Switching System (VSS) 1440 platform fully supports the use of Cisco integrated service modules such as the Cisco Application Control Engine (ACE), Firewall Services Module, and Network Analysis Module. The Cisco Catalyst VSS is used as a network-based services platform for the enhanced secure multi-tenant architecture.

Cisco Firewall Services provide stateful firewall security capabilities within the SMT architecture. Integrated as a module or as a dedicated appliance, the virtual Cisco firewall security contexts may be transparently introduced at the Layer 2 network level or as a router "hop" at Layer 3. With either deployment model, the security policies associated with each virtual firewall context are consistently applied to protect the related networks.

The Cisco Application Control Engine (ACE) module and application platforms perform server load balancing, network traffic control, service redundancy, resource management, encryption and security, and application acceleration and optimization, all in a single network device. The Cisco ACE technologies provide device and network service-level availability, scalability, and security features to the data center.

The Cisco ACE offers the following device-level services:

  • Physical redundancy with failover capabilities
  • Scalability through virtualization allows ACE resources to be logically partitioned and assigned to meet specific tenant service requirements
  • Security services including ACLs and transport encryption (SSL/TLS) between the ACE virtual context, client population, and associated server farm

The Cisco Intrusion Prevention System (IPS) appliances are network sensors that may be positioned throughout your infrastructure as promiscuous network analysis devices or inline intrusion prevention systems. The Cisco IPS sensors protect you by detecting, classifying, and blocking network-based threats using attack signatures associated with worms, viruses, and various application-abuse scenarios. This process occurs on a per connection basis, allowing legitimate traffic to flow unobstructed.

The Cisco Network Analysis Module (NAM) comes in several form factors:

  • Integrated service module for the Cisco Catalyst 6500 switching platform
  • Physical appliance with multiple Gigabit or 10 Gigabit Ethernet support
  • Virtual service blade for Cisco Nexus 1000v deployments

Regardless of the model, the NAM offers flow-based traffic analysis of applications, hosts, and conversations; performance-based measurements on application, server, and network latency; quality of experience metrics for network-based services; and problem analysis using deep, insightful packet captures. The Cisco NAM includes an embedded, Web-based traffic analyzer GUI that provides quick access to the configuration menus and presents easy-to-read performance reports on the Web for different types of services and traffic. The Cisco NAM line of products improves visibility into and monitors the performance of the many physical and virtual layers within your infrastructure.

Cisco Security Manager is an enterprise-class management application designed to configure firewall, VPN, and intrusion prevention system (IPS) security services on Cisco network and security devices.


While we've only scratched the surface of the complete capabilities of our enhanced SMT architecture, this introduction hopefully gives you a good overview of its security capabilities. To learn more about secure separation capabilities plus new and improved capabilities for the other pillars—availability, service assurance, and management—check out our comprehensive design guide. (Note that this guide refers to enhanced SMT as ESMT; they are the same thing.) This 140-page guide covers every aspect of SMT architecture and deployment.

Got Got opinions about Enhanced SMT?
Ask questions, exchange ideas, and share your thoughts online in NetApp Communities.

Chris O'Brien
Solutions Architect

Chris architects the latest data center technologies in Cisco's Systems Architecture and Strategy group. He is currently focused on data center design validation and application optimization. Previously, Chris was an application developer and has been working in the IT industry for more than 15 years.

David Klem
Reference Architect

David joined NetApp in 2005 as part of the original team that architected and built the NetApp Kilo Client, a 1,700-node internal cloud used by NetApp Engineering. David now works for the NetApp Infrastructure and Cloud Enablement team, focusing on best practices and solutions for cloud-based architectures. Before joining NetApp, he was at Cisco for six years, where he worked as an engineer on the Cisco Catalyst® switch platform.



This NetApp Community is public and open website that is indexed by search engines such as Google. Participation in the NetApp Community is voluntary. All content posted on the NetApp Community is publicly viewable and available. This includes the rich text editor which is not encrypted for https.

In accordance to our Code of Conduct and Community Terms of Use DO NOT post or attach the following:

  • Software files (compressed or uncompressed)
  • Files that require an End User License Agreement (EULA)
  • Confidential information
  • Personal data you do not want publicly available
  • Another’s personally identifiable information
  • Copyrighted materials without the permission of the copyright owner

Files and content that do not abide by the Community Terms of Use or Code of Conduct will be removed. Continued non-compliance may result in NetApp Community account restrictions or termination.