Questions about encryption

[gmilazzoitag]

Hi,

I know that nowadays for NetApp FAS and E-Series there are two different ways to encrypt data.

Use a subset of disk with built in encryption or adopt the Safenet solutions. 

I've a deep experience with old Decru Datafort appliances so I can easily understand what Safenet does, both for FC SAN and for NAS environment.

My first question is this.

Using HDDs or SSD with built-in encryption do not exlude every device on the same SAN/NAS to read data in clear format, is it? I mean, after all, are disks that protect me in case my shelves are stolen but nothing more or am I missing something I don't know?

If I think to Safenet solutions instead I'm sure that only people/application allowed to perform I/O through Safenet devices (put between reader/writer and volumes) can read those data. This is real protection and encryption. In somebody else that them try to access and, unluckly, phisycally reach those contents, this will read "garbage"...

With disk with builtin encryption instead?

I hope I've been able to expose the questions.

Regards.

[bobshouseofcards]

As part of a total environment refresh project I've been involved with, to include new encryption capability, I've been deeply involved with trying to understand the data encryption options for both FAS and E-series.  Here is some information I hope helps you.

First - both storage platforms rely on full disk encryption (FDE) capable disks.  Data is not encrypted external to the disk drive itself - this is truely data at rest only.  Once in the controller or on the network, data is not encrypted.

The nice things about FDE disks is that since the encryption engine is built into the disk itself, all encryption takes place at or near wire speed to the disk.  Hence there is no single bottleneck to performance and there is no expectation that an encrypted solution will perform differently than an unencrypted solution using the same basic hardware.  Granted, the encryption capable disks will cost more, and there are other add-ons for both platforms as well.  But, since controllers don't see encrypted data, all controller based features just work.  This is significantly important for FAS controllers to allow DoT features like compression and deduplication to be transparent.

FDE disks require a key to be generated and loaded down to the disk to enable encryption of data on the way to the disk platters and to decrypt on the way from the disk platters.  Here is where the platforms diverge.  E-Series platforms with the encryption license option allow an administrater to manually set (generate) a single encryption key which is used for all the disks.  Additionally, E-Series only requires basic FDE capable disks which are available in a ton of sizes from 600GB -1.2TB performance, 800GB and up SSDs, and the usual capacity sizes up to 6TB disks.  Very flexible if it meets the needs to, generally, protect data when a disk goes out of service, for instance, or from someone stealing a disk or a shelf.

The FAS platform takes a different take.  Rather than just encrypt on FDE disks, the FAS encryption platform is FIPS 140 standard compliant, for government use and for other regulated industries.  This adds a lserious layer of complexity.  For starters, the disks themselves must be certified to the FIPS standards which means the drive needs to hardend against intrusion, should destory itself if penetrated, and should show external indicators if physical intrusion was attempted.  At the platform level, each disk gets a separate encryption key, and the keys need to managed by a FIPS compliant key manager.  That is where the SafeNet devices come in - they aren't encryption platforms, they are key managers.  All they store are the keys to the drives.  When an encrypted FAS platform starts, which NetApp refers to as NetApp Storage Encryption (NSE), the controller knows as it boots that it is in encrypted mode, contacts the key manager securely using SSL with known certificates and key pairs at a known address over a specific known port again all set as options before DoT boots, downloads the keys for all the disk drives, and pushes those keys out to each drive.  Only then does it start accessing the data on the disks because once the key is reloaded on each disk the data can be decrypted.

The FAS solution is arguably more secure and meets certain high end certifications that allow it to be used in regulated areas (government, industries) around the world.  It is also massively more complex and expensive to deploy and operate.  For example, there are only a limited set of FIPS compliant disk drives comonly available from multiple drive manufactureres.  Hence, in an NSE solution you are limited to 900GB SAS disks and 800GB SSDs in a DS2246 and 4TB NL-SAS in DS4246 shelves.  Note that you can't use DS4486's if you want density.  Further - if a FAS controller (single or HA pair) uses encryption all disks attached to the controller must use encryption.  Spare replacements require manual intervention to set a key for the disk - no autoassign at controller/shelf level and it's just ready as a spare.  And the officially supported Safenet appliances are really expensive comparatively and of course you need at least two to protect the keys - lose the keys and you're out of luck.  And during upgrades and hardware maintenance (like motherboard replacements, head replacements, etc.) there are all kinds of extra steps you need to consider to ensure you don't lose the relationship between the controllers, the key managers, and the disks.  But if you need that FIPS certification, you need it and at the moment from NetApp there isn't any other choice.

The E-series is more flexible.  Obviously there are more disk options.  I *believe* that you can encrypt some of the disks but not all if you've deployed mixed disk types, but I think it's all or none within a single shelf unit.  Replacements are automatically set for encryption since the controller holds the key.  New controller, lost configuration?  Just restore the single encryption key.  So way easier - just not going to meet the FIPS standards.  And of course E-series is block protocols only.

One mechanism to leverage easier encryption and full DoT capabilities is to use E-Series with encryption behind FAS heads with the FlexArray (formerly the old VSeries controllers, now just a license option on FAS).  When considering encryption we speced it out both ways - performance was not an issue (the E-Series are really very good at what they do block wise).  Total rack space for our example was significantly less because we could use both higher density shelves and higher density disks on the E-Series so at scale the differences got where pronounced quickly.  

NetApp does not offer any product similar to the old Decru Datafort that encrypts in transit on the wire.  In fact it doesn't make sense for them to do so with the advent of FDE capable disks.  Rather all encryption is not a part of the storage platform rather than being external to it.  Practically speaking within the data center at least encryption in transit (network/SAN) is less of a concern anyway.  If someone breaks into your data center to apply a tap on the wire to intercept data, or if one of your employees does it internally, you have a larger problem that basic encryption is not going to solve.  If someone gains access to servers, well then they have credentials to potentially request the data on the server in a decrypted form.  Again - a different problem.  Unless the data is encrypted up to the point where it is displayed to the target user, data is vulnerable somewhere in your system to anyone who can enter with the right access or credential.  Even in a completely end to end encryption mechanism, data remains vulnerable to someone with an unauthorized credential.  Encryption may reduce the potential attack vectors, but as you add more you add cost and complexity without necessarily reducing the primary attack vectors (consider Sony - an inside job).

Hope this helps you while considering encryption.

Bob Greenwald

Lead Storage Engineer | Huron Legal

Huron Consulting Group

NCDA | NCIE-SAN Clustered Data OnTap

[gmilazzoitag]

Hi Bob,

first of all thank you very much for the time you've spent to write this long and detailed answer. Really good points.

Coming back to the argument I read "NetApp does not offer any product similar to the old Decru Datafort that encrypts in transit on the wire." Do you mean that NetApp does no more offer?
I remember, a couple of years ago, a project of technology refresh for a customer of us using Datafort (that has gone EOS) where we've tried to swap with Safenet solutions (a cluster of cryptainers -let me call in this way- appliances and key manager one).

Anyway, I agree with you that hole in the security can always be present at any level but personally I still continue to think that encryption over the wire represents another "firewall" between data and potential intruders. I don't know what Safenet does now but I can remember that in case of tape backup Decru offered also a way to store encrypted data on tapes themselves: I *know* your potential answers: there are tape libraries with encryption 😉

Regards,

Giacomo Milazzo

Sinergy SpA

[bobshouseofcards]

NetApp does not offer any Decru equivalent gear.  The DataFort product was killed off a while ago.

In the NetApp FAS based encryption world, SafeNet and NetApp partnered on encrypted storage to advance that solution.  They both work in the KMIP standards on key exchange, and technically, NetApp product literature advertises that NetApp NSE (FAS) based systems will work with compatible KMIP standards key manager. NetApp officially supports only SafeNet key managers though despite the product information available on their web site.  During their OEM arrangement with IBM (IBM N-series storage is rebranded NetApp) they also supported IBM's Tivoli key manager solution which is also KMIP compatible.  In theory any KMIP system should work, but don't look for support and the mechanism is cumbersome enough from documentation that you probably want a supported solution.

SafeNet as a company may offer wire based encryption products - I've never investigated them beyond their key managers.  Because their key managers are KMIP compliant they should work with any other KMIP based encryption system, say for instance as the key manager for an encrypting tape library (since you mentioned them :-).  The one SafeNet SKU that NetApp resells is a high end appliance (with a high end price) that for all but the most massive NetApp installations is overkill.  How many million disk NetApp farms do you see for instance, yet that's how many keys a single NetApp sold SafeNet key manager can support.

There certainly are devices out there to encrypt on the wire at wire speeds - Riverbed I think has some among others - but as the wire speed goes higher the cost goes up quickly too.  I think that's why there is more focus on the encryption at key points - WAN devices, disks, tape drives.  The capability to add encryption at those key end points is easier because there is a known max bandwidth to handle, there are now robust key exchange mechanisms allowing automatic key generation and separation of concerns, and there is vendor and product line interoperability based on recognized standards.  In the time when the Decru products were live, most encryption solutions were integrated and/or isolated within a specific product and vendor and you were quickly tied into a particular solution.

I sincerely wish, especially where NetApp advertises that FAS NSE solutions work with any compatible KMIP based key manager that they'd actually stand behind it.  Then again I wish and hope they make the entire NSE based solution simpler in general.

[JoeO]

Hi Bob, great information!  There seems to be very little expertise on the FAS NSE solution, so again thanks for the info.  

One quick question (I think I know the answer but will ask anyway)...if a client has moved data onto a NetApp NSE system prior to turning on encryptions (data in the clear), is there any way ro turn on the encryption and save the data that is currently in the clear?  Or am I looking at migrating the data off then back on after turning on the encryption?  NeoScale, back in the day, had a 'Data Prep' utility that ran against the clear data and allowed us to encrypt a LUN on the fly through the appliance.  I haven't read anything like that for the NSE solution but I will await your answer.

Thanks...

[bobshouseofcards]

Joe -

That is an open question for me too.  In theory there should be a way to do that.  For instance, you have to manually key a disk new disk on hot spare replacement, so obviously a disk can be in the system before it is keyed.  However, it is also true that all disks are either NSE or not NSE in an encrypted solution, so once you turn on NSE it might not be possible to use disks that are not NSE'd.  And, I believe there is support to "rekey" a disk, which make sense in the case a key is somehow compromised (not going into how that might happen).

I've had the same queries out to my NetApp support team and a solid firm answer is yet to come out.  I suggest though that you NOT depend on that behavior being available - that is you will need to do a volume move migration type event on data to get it encrypted, and you should not try to key a disk that has active data.  The information I'm getting leads in that general direction (either not "supported" or "in theory yes but don't depend on it working" type of answers).

[JoeO]

Thanks Bob and I agree with your position on not depending on a reliable method to encrpt existing clear text data.  When I did it in the past it was always an anxious few hours waiting on the results.  Here's what I had in mind, let me know what you think.

There are two filers, Primary and DR, Primary is SnapMirrored to DR.  Unfortunately, moving the DR filer to the Primary site to accelerate my process is not an option.

• Full backups, both sites if possible.
• Pause SnapMirrors
• Delete Volumes at DR site and turn on NSE Encryption
• Recreate Volumes and resume SnapMirrors creating new encrypted baselines.
• Once everything is back in sync do another full backup
• Depending on time required to create baselines decide on taking an outage on Primary or go into DR mode and promote DR filer
• Pause Snapmirrors
• Delete Volumes at Primary site and turn on NSE Encryption
• Recreate Volumes and reverse SnapMirrors to bring DR data back to Primary filer
• Once data is rebuilt reverse the SnapMirrors to original configuration

Certainly Task Lists with full details must be put in place but this the general SOW I was going to propose.

Thanks... 

[bobshouseofcards]

Joe -

Yes that would certainly work, with the obvious caveats of time involved, lack of "DR" during the recreates, and any access penalties during the run from the DR side as opposed to the primary, but I'm not telling you here anything you don't already know.

Questions - do you already have NSE capable disks and the key manager appliances installed or planned?  For instance had you purchased only NSE capable disks at some point in the past to ensure you could go to full encryption at a future point?  Just curious as to how you addressed that point, as I'm in a place where I have to make similar future proofing decisions on hardware that is ready for replacement.  In my case, unfortunately due to how my systems have grown, I don't get to replace an entire cluster at one time, but only parts of multi-node clusters as hardware falls off the accounting books.  So I have to decide whether to start buying NSE capable gear as I replace, and perhaps flesh out the key manager later in an appropriate budget cycle and then encrypt, or bite the bullet on the key manager now when it wouldn't be fully utilized for several more years.

That key manager piece isn't cheap.

[JoeO]

Hi Bob,

This is actually for a client of mine.  Apparently they bought the NSE solution without the key manager and began using the filers without encryption being turned on.  I was brought in initially for the THALES keyAuthority piece but since I'm so familiar with NetApp I am proposing the whole SOW to get them from clear text storage to the full NetApp NSE solution with keyAuthority.  Doesn't appear to be a simple transition from where they are now to encrypted Data-At-Rest.  From what I see it has to be a data migration off the filer and then back on after turning on encryption.  My only option to speed things up and keep their exposure to a minimum is to come in with some swing equipment, maybe NetApp can help??

Thanks,

JoeO

[bobshouseofcards]

Ooo - fun one for you!

Yup - I agree that some sort of swing is going to be needed.  You couldn't even just add some generic shelves to the existing controllers unless they were all NSE capable, as an HA pair is either fully encrypted or not.  If the customer has a good enough relationship with NetApp perhaps they can arrange some temporary/loaner stuff, but then again I also can't imagine who a NetApp sales team would have let a customer get into this position.