NVMe - A Step into the Future

By Keith Aasen, Solutions Architect, NetApp


This fall, NetApp completely refreshed its FAS hardware line. Although some might argue that proprietary hardware is a thing of the past, I would argue that plenty of innovation remains in the process of assembling hardware systems out of industry-standard components. A prime example is the addition of Non-volatile Memory Express (NVMe) devices on the newly released storage controllers. I thought I would take a bit more of a techie deep dive into the NVMe world, how we are using it today and where we go with it in the future.


To fully appreciate NVMe devices, we need to go back in time. Although there were multiple CPU-to-storage protocols in the early days of computing, one that gained popularity and therefore standardization was the Small Computer System Interface (SCSI). This standard allowed efficient communication between a computer’s CPU and the locally attached disk drives.


When storage arrays came along, we needed a means to attach large numbers of drives to the CPUs while maintaining use of the SCSI standard. From this need, Serial Attached SCSI (SAS) was born. This development allowed continued use of the SCSI command set while attaching numerous drives to a shared bus.


And so it was for the past couple decades. The actual drive technology could be changed (SATA or SAS drives), but the bus and command set remained the same. The addition of flash solid-state drives (SSDs), however, has prompted change. You can place an SSD onto a SAS interface and bus (which most vendors, NetApp included, do today). But the 550MBps to 600MBps limit on a SAS interface is a bottleneck to the SSD, which can deliver much more.

SCSI Commands

NVMe removes this bottleneck. By using a new set of commands, a massive number of I/O queues, and direct connection to the CPUs through the PCI bus, an NVMe-connected SSD can push upward of 4000MBps. That’s quite a boost over the 550MBps on a SAS interface.

PCI Express Connected

Such an improvement almost always results in a massive technology shift, and NVMe will be no exception. But there is one final hurdle. Today the NVMe standard requires the direct connection of the drive to the system’s PCI bus. That’s fine for a laptop but is not ideal when designing a large storage system. We need a means to place these drives on a shared bus without losing any of the massive bandwidth and incredibly low latency (6us on average). Several standards are in the works, for example, RoCE and iWARP using converged Ethernet and FC-NVMe using Fibre Channel. NetApp is an NVMe promoter (oddly, one of the few storage companies on the project), and as such we have direct access to the protocols as they are ratified. As soon as they are ready for prime time, we will be there.


NVMe Commands


So, is NVMe only a promise of a future technology? Hardly. As I mentioned, NVMe is available today on system-connected drives, and so all the new NetApp® systems (FAS2600, FAS8200, and FAS9000) come with an NVMe-connected drive.


FAS 2600


We use this drive as a NetApp Flash Cache™ device, accelerating the entire storage system. The massive I/O capability and low latency allow the device to accelerate storage workloads well beyond what the underlying media could deliver.


NVMe is also the protocol that will open up usage of the next-generation storage medium that is known as storage class memory. To recognize the benefits of this improved medium, an ultralow-latency protocol is required. Having these ports on the new storage controllers opens up possibilities to accelerate even all-flash arrays into new realms of performance without a complete change of the underlying media.


While other vendors are making noise about NVMe, NetApp is deploying it today and is working with the industry to make it a standard for tomorrow.


Nice read on this new technology! 

Thanks Keitha for this well written blog.

Good stuff )

Hi , I just got some confused.

Lots of people said NVME module as new generation instead of of PAM card.

So do I still need to tune or is there related options as PAM had before? (ex: options flexscale.enable ... )



We are using the NVME drive and connector for the new FlashCache (formally PAM). That is on the hardware side. On the Software side (in ONTAP) you can still tune the FlacheCache as you did before. In fact in ONTAP 9 the are additional tuning options you can use if you have a specfic workload you need accelerated or you can leave it at the default which does an excellent job of accelerating more workloads. The only difference is now that the FlashCache is connected via NVMe we can access data in the FlachCache at a faction of the latency allowing cache hits to be served in microseconds and with less CPU impact to the system. I hope that helps.

Hi, Nice read.


Could you please tell me for 8200 model the NVMe card is already built-in as a FlashCache device or it needs to be plugged separately ? 



Great stuff Keith!


@ShashiNetApp-Community Good question. On the 8200 the NVMe module is included by default but can be removed from the config if you want. I call it a module since it does not consume a cardslot like the old FlashCache card did.


Great article Keitha, shared it on LinkedIn.  Keep more coming!!


Hey Keith,

       great post as usual, however I think theres one speeds and feeds metric where I think you might have made a mistake when you said ..


But the 550MBps to 600MBps limit on a SAS interface is a bottleneck to the SSD, which can deliver much more.


The throughput on dual ported SAS disk is about 1200 Megabytes per second per lane for an aggregate theoretical throughput of about 2400 Megabytes per second, and there are dual ported SAS SSD's benchmarked doing about 1900 megabytes per second.


Still not as good as what you see out of an SSD using NMVe across four or eight PCIe lanes, but pretty good nonetheless. To be honest I was pretty surprised when I found this out during my research on NVMe which I documented in some blog posts on linked in .. the most relevant one is here https://www.linkedin.com/pulse/how-cool-nvme-throughput-john-martin