topic Re: EMC vs NetApp Deduplication - Fact Only Please in ONTAP Discussions

EMC vs NetApp Deduplication - Fact Only Please

wade_pawless — Thu, 05 Jun 2025 07:07:08 GMT

I'm writing this in hopes to become better educated on the facts, not opinions, about the pros and cons as well as similarities and differences between NetApp's "current" Deduplication technology and EMC's "current Deduplication technology.

I have deployed NetApp solutions within my previous environment, but my current (new workplace) utilizes EMC. On a personal note, I prefer NetApp hands down. However, my responsibility is to define the capabilities and make decisions from the facts.

I have read some about EMC and NetApp fighting over Data Domain around 2009. When talking to EMC, they're deduplication recomenations are Data Domain.

EMC claims that their Data Domain product provide real time deduplication at a 4kb block level which they claim to be much more efficient. EMC mentioned that NetApp does dedupliation at around 128kb. THe bottom line is a claim to better performance capabilities. Thoughts?

Does the curent version o ONTAP utilize SIS or Deduplication?

EMC claims SIS is a "limited form of deduplication". http://www.datadomain.com/resources/faq.html#q5

Please clarify the facts regarding the type of deduplication utilized by NetApp as well as any thoughts to the comments above. Facts only please.

As a side note. I'm currently comparing the NetApp VSeries with EMC produt lines. The goal is to place a single device in front of all SAN's to provide snap shot and deduplication to all SAN data. Over time we'll be bringing multiple storage vendors into our environment. The VSeries is a one stop shop. We can vertualize all data regardless of vendor, provide snap shot capability, dedupe the data, and simplify management. EMC's solutions requires me to purchase two new devices, a EMC VG2 and a EMC Data Domain. Event with EMC's recommendations, there is no capability to snap shot other vendor data. The Data Domain appliance will only provide dedupe for all vendors. The EMC VG2 is being recommend to consolidate multiple file storage servers into CIFS as well as provide NFS for virtualization. So, in the end EMC is saying buy one appliance from us to provide dedupe, buy a second appliance to provide NAS capabilities. Wait a minute, all of these features are built into a single NetApp.... Thoughts?

Re: EMC vs NetApp Deduplication - Fact Only Please

aborzenkov — Wed, 13 Oct 2010 05:09:37 GMT

NetApp deduplicates on 4K block level - this is fact. AFAIK DataDomain is using variable block length.

Does the curent version o ONTAP utilize SIS or Deduplication?

The feature is officially called A-SIS, but everyone is using just Dedupe speaking about it. Both terms are used interchangeably as far as I can tell.

EMC claims SIS is a "limited form of deduplication". http://www.datadomain.com/resources/faq.html#q5

NetApp A-SIS works on block level, not on file level. So the above simply has nothing to do with NetApp.

Re: EMC vs NetApp Deduplication - Fact Only Please

radek_kubka — Wed, 13 Oct 2010 07:22:17 GMT

Well, to me the key differentiator is that NetApp de-dupe can be (& often is) used for primary data (e.g. VMware datastores), whilst Data Domain is aimed purely at secondary data (i.e. backup images) - unless they changed their positioning recently.

Regards,

Radek

Re: EMC vs NetApp Deduplication - Fact Only Please

kevingolding — Wed, 13 Oct 2010 09:29:15 GMT

Netapp Called their Deduplication Advanced Single Instance Storage, which means that ASIS IS Deduplication.

ASIS can be configured on either primary data or secondary storage volumes and is agnostic to the type of data in those volumes. There is only one other vendor (unless things have changed recently) that can provide DEDUPE on primary data, everyone else does it on the secondary data (backups).. so not that useful really.

Deduplicated folders are mirrored (using SM) in a dedupe state so replication is more lightweight.

ASIS runs at the block level, so Word Documents and Excel Documents don't matter, it doesnt give a monkeys what the blocks are part of.

It isn't perfect, but reducing the primary storage footprint, to my mind is what we want to talk about. Who cares if the backups are smaller, we want more usable space out of our terabytes...

Re: EMC vs NetApp Deduplication - Fact Only Please

martinmeinl — Wed, 13 Oct 2010 11:54:44 GMT

Hi all,

before you validate a solution or technique, please consider in your decissioins, what are you looking for?

Do you need realtime Deduplication, post (async) deduplication, will primary data be the target or backup data?

A-SIS works only job driven and it consumes a lot of Filer-Ressources via deduplication run.

Via Deduplication run the Filer is degraded in performance.

Using A-SIS for Backup-Data will only show post processed the reduction of space (so primarly it consumes all the space befor reduction)

DataDomain from EMC is Realtime Inline DeDuplication and is targeted for secondary (backup) data and optimized for this flavor of data.

DataDomain in this case only consumes reduced capacity onthe backend and saves large amounts of capacity in the Storage System.

Using DataDomain as Online Storage will show weak perofmrance in transactional IOs, but good performance in streaming IOs.

EMC also delivers Online Storage DeDuplication in their Celerra Products Code for Fileservices only, and also uses a Post-DeDuplication process (automated process)

EM;C Combines also Compressioin with this to gain better Reduction as they only use Single Instancing.

Benefit out of this concept seems to be lower consumption of COU Ressources running the DeDuplication.

So in fact it's not easy choosing the right vendor and concept as you first need to concentrate on you benefits of each solution and method and the Tradeoffs of this concept.

You never will get all-in-one from any vendor at all--> you never can be left and right once (unless you get cloned) 😉

Re: EMC vs NetApp Deduplication - Fact Only Please

inx_russel — Thu, 14 Oct 2010 17:18:32 GMT

A key differentiator between NetApp and most other dedupe options is how they determine data is duplicate or not. Most vendors use "safe enough" hash based matching, while having false positives is rare as the amount of data being deduped increases so does the risk of hash collisions. NetApp, being post-process, can afford a more complex matching alogorithm. NetApp only uses hashes to detect /possible/ duplicate data, it then does a comparison of those possible matchines byte by byte...which is far safer.

Re: EMC vs NetApp Deduplication - Fact Only Please

amiller_1 — Fri, 15 Oct 2010 03:40:02 GMT

So....there's lot of answers here given the varying array of products...no "ONTap to rule them all model" (and I don't claim to know them all...just the general topics).

For primary storage....

Symmetrix and/or V-Max -- nope, no dedup.
Clariion -- nope again.
Celerra -- yep and compression too but there's different variations that I frankly don't understand. Googling on "celerra deduplication" seems like the best path....3rd hit is an EMC white paper....albeit 7 months old.
- If you do have a Symmetrix or Clariion and have it front-ended with a NAS gateway (i.e. what a Celerra really is....a Celerra = Clariion + NAS gateway bundled together), there may be some dedup possibilities on the NAS protocol side.

For backups....

Data Domain -- yep...good tech IMHO and if not for that little bidding war would have been a NetApp product after all.
Avamar -- yep again....source side dedup which has some interesting implications.

I'm sure there's more out there but I can't claim to be an EMC expert....I just try to make sure I know the general lay of the land though.

Re: EMC vs NetApp Deduplication - Fact Only Please

chriszurich — Fri, 15 Oct 2010 13:53:34 GMT

One Dedupe drawback I've run across while reading and learning about NetApp (I'm a former EMC guy...) is that deduplication on a volume which has snapshots turned on negates the dedupe savings. I've read that the following happens:

Scenario

Volume created and populated with data (LUN or NFS/CIFs share etc.). As the data changes snapshots are generated for changed blocks. Lets say that deduplication is then turned on after a certain period of time. At the point where deduplication is turned on all of the blocks which are consolidated via the dedupe process are saved within the snapshot performed prior to the deduplication process being kicked off. This essentially negates the decuplication savings since the deduplicated blocks still exist on your primary storage within the prior snapshot. In order to fix this conundrum you must delete all snapshots performed before the deduplication process is kicked off. This of course has the drawback of destroying your recovery capabilities for that volume.

Can anyone refute this? I see this as a huge flaw in the NetApp primary storage dedupe solution since data integrity/recovery is the primary goal of most production storage solutions with deduplication and optimization as a close second.

Re: EMC vs NetApp Deduplication - Fact Only Please

radek_kubka — Fri, 15 Oct 2010 15:33:46 GMT

That is correct, as currently snapshots are not 'de-dupe aware'. Whether it will change in the nearest future is beyond my knowledge.

This is arguably a drawback, but not a show-stopper IMHO:

- if you don't de-dupe a volume with existing snapshots, you are not gaining anything

- if you de-dupe a volume with existing snapshots, you are not gaining anything until your 'pre-de-dupe' snapshots expire & get deleted.

Ideally de-dupe should be run when you load a lot of data into a freshly created volume with no snapshots & after that you start taking snapshots (e.g. as a SnapMirror baseline).

Yes, it means a bit of a jiggery here & there, but at the end of the day NetApp ASIS is the only widely deployed primary data deduplication operating on block-level.

Regards,
Radek

Re: EMC vs NetApp Deduplication - Fact Only Please

chriszurich — Fri, 15 Oct 2010 18:03:26 GMT

Thanks for confirming. I thought of the second solution you mentioned after I finished that post. It's not a huge deal breaker since as you mentioned you just need to wait for the pre-dedupe snapshots to roll off.

One additional argument against dedupe on primary storage which I've seen within my environment is that dedupe processes take up production resources. As our environment has grown we've had to redistribute dedupe jobs so that CPU utilization doesn't spike into the 90-100% range. It's yet one more thing to monitor to ensure the "optimization" built into Data ONTAP doesn't undermine our core capability of serving I/O in a timely manner to hosts.

Re: EMC vs NetApp Deduplication - Fact Only Please

amiller_1 — Sat, 16 Oct 2010 02:25:30 GMT

Twould be wonderful if it were a non-issue...but practically speaking is usually feasible to schedule the dedup runs before the snapshots (at least when doing daily snapshots). Just did that earlier today actually while finishing up a project (enabled dedup on some new volumes and then a nightly snapshot schedule 2 hours after the dedup run).

Re: EMC vs NetApp Deduplication - Fact Only Please

mheimberg — Tue, 19 Oct 2010 09:00:07 GMT

This is something I will never understand:

....

The VSeries is a one stop shop. We can vertualize all data regardless of vendor, provide snap shot capability, dedupe the data, and simplify management.

Buy a NetApp because of its (obviously) undisputed simplicity in management, superior snapshot-capabilites, dedupe for primary *and* backup data, not to forget the true unified architecture...but the data is not trusted to be hosted on NetApp storage, instead you give away a lot of potential in further simplifying management, backup and restore, update management etc.

Why do people go only half the way? This will always be a mystery to me.

Mark

Re: EMC vs NetApp Deduplication - Fact Only Please

thomas_glodde — Tue, 19 Oct 2010 11:22:37 GMT

Because you might have spend a few grants on your old storage and kinda want to "reuse" the hard disks. Then you might use a V-Series.

Besides, V-Series is sometimes used NAS only on existing high-end FC SANs to properly cover fileservices.

Re: EMC vs NetApp Deduplication - Fact Only Please

mheimberg — Tue, 19 Oct 2010 13:22:14 GMT

Because you might have spend a few grants on your old storage and kinda want to "reuse" the hard disks. Then you might use a V-Series.

Yes, of course. But sooner or later your disks are end of warranty, EOL and out of support. Then comes the time when I am glad that I can attach also NetApp disk shelves to a VSeries and smoothly migrate to a NetApp-only environment.

The point is, Wade wrote: "Over time we'll be bringing multiple storage vendors into our environment" - it is this multi-vendor-multi-anything-thing I decline.

But of course, there might be companies that really need the "best of breed" in any domain - because I have none in my portfolio doesn't mean they don't exist. But you may also agree that sometimes customers tend to exaggerated solutions, and then there must be someone questioning the approach....

Besides, V-Series is sometimes used NAS only on existing high-end FC SANs to properly cover fileservices.

To me this is a very funny point: you need a NAS access, but unfortunetaly you only have this super powerful FC SAN in your datacenter because the sales of this FC SAN always told you, you will never need NAS and this whole "unified architecture" babble is only marketing cramp from competitor? HeHe, good do we have VSeries! And then...see above...

Mark

Re: EMC vs NetApp Deduplication - Fact Only Please

thomas_glodde — Tue, 19 Oct 2010 13:39:32 GMT

Markus,

you are preaching to the evangelist 😉

Its sort of nice entry tho as you can say to the potential customer you can have the features NOW and you can sell the disks later, kinda smooth migration.

IHAC who sold his soul to HP for Clients, Servers, Switches, Printers and Storage. He is not willing to add NetApp storage as he doesnt have to bother with compatiblity matrix and 2 companys case finger pointing when adding SAN applications, regardless of 0h recovery functions etc. But we could convice him to at least buy a V-Series for proper NAS functionality as well as to give the unique NetApp features at least a try in his SAN environment.

Re: EMC vs NetApp Deduplication - Fact Only Please

wade_pawless — Tue, 19 Oct 2010 16:07:09 GMT

First, I would like to say that I appreciate all of you taking a moment to reply and share the knowledge with me. Since the posting of this thread I have continued to work with EMC as well as NetApp to further define the architecture requirements to facilitate storage virtualization enabling centralized snap shot, deduplication and storage consolidation while maximizing ROI regardless of back end vendor flavor.

The bottom line:

NetApp- regarding architecture simplification as well as capabilities, NetApp wins. However, cost comparisons have not yet been completed.

In a scenario where you have a primary site and secondary site which will become your DR, then a hot site later, NetApp requires less hardware.

(NetApp’s Solution) To provide snapshot, deduplication, consolidate file storage via CIFS, leverage fast backup library technology (SnapVault), leverage NFS for virtualization storage, as well as enable all of these capabilities on existing SAN’s and provide all SAN capabilities in a unified architecture, only one appliance appears to be necessary , a VSeries. If you want all of this data sent offsite, you can place a lower line NetApp FAS such as the 3100 at your DR site. In total we’ve added two appliances as well as some storage to the DR site correct?

(EMC’s Solution) To facilitate CIFS and NFS as stated above, EMC recommends the VG2 which essentially adds a NAS appliance to the architecture. To facilitate deduplication, EMC refers you to their Data Domain product line which adds another appliance to the architecture. Lastly, EMC’s answer to my requirement of leveraging existing other vendor storage to maximize ROI and prevent acquisition of storage that is already in place was…. we’d look into your current EMC capacity and look for areas to expand capacity such as upgrading 1TB drives to 2TB. The bottom line is EMC would require that I purchase more EMC storage to replace other vendor storage. Now, if you want to continue this path into a DR site you’ll be doubling the hardware.

So,

Onsite: 1 VG2 +1 Data Domain+Additional Storage+(EMC SAN already in place)

DR Site: 1 VG2 +1 Data Domain+ New EMC Storage Processor (or head) in NetApp terms + additional storage

Additionally, EMC’s solution will only provide snap shot capabilities. The result is more storage consumption for backups. Since we could not snapshot, we’d be forced to rely on an application level backup management product adding more layers, complexity and costs associated with licensing.

Remember I said facts only please, so let’s list the downside to the VSeries. EMC informed me that storage virtualized through the VSeries has a 30% loss in capacity. I spoke to NetApp whom informed me the number was 20%. Any thoughts or info on this are welcome.

Thanks again, please let me know what you think.

Re: EMC vs NetApp Deduplication - Fact Only Please

chriszurich — Tue, 19 Oct 2010 19:04:07 GMT

Here are the NetApp Data ONTAP overhead numbers I'm aware of.

WAFL overhead = 20% (no way to get around this)

Aggregate snap reserve capacity = 5% (can be changed)

Volume snap reserve capacity = 20% (can be changed)

Re: EMC vs NetApp Deduplication - Fact Only Please

brendanheading — Tue, 19 Oct 2010 19:53:46 GMT

This seems like as good a place as any to pose an interesting question.

What's the hardware difference between a v-series and the corresponding FAS ? eg what's the difference between a V3100 and a FAS3100 ? Surely the FC interconnect between the controllers and the shelves is the same, especially given that the V-series can use native disk shelves.

I can understand why NetApp would want to carefully control the connection of third-party shelves but surely this can be accomplished with a license key ?

Re: EMC vs NetApp Deduplication - Fact Only Please

aborzenkov — Wed, 20 Oct 2010 05:11:46 GMT

WAFL overhead (reserve) is 10%. Where does 20% come from?

Re: EMC vs NetApp Deduplication - Fact Only Please

chriszurich — Wed, 20 Oct 2010 11:23:11 GMT

I got the 20% WAFL overhead number from a recent NCDA training I attended. Where are you getting 10% overhead from? Perhaps someone from NetApp can weigh in?