topic Re: Most unusual application of dedupe? in ONTAP Discussions

Most unusual application of dedupe?

friea — Thu, 05 Jun 2025 07:35:02 GMT

lfreeman mentioned that people are using deduplication in ways NetApp didn't originally expect. What's the most unusual application of dedupe anybody's aware of?

Re: Most unusual application of dedupe?

reinoud7 — Sat, 08 Mar 2008 22:49:49 GMT

Hi Friea,

It's not realy unusual, but here is the list of applications where we use today dedupe. Let me know if you find those applications unusual!

dumps of VMWare
dumps of SQL and Sybase
our share with the sources of all our software
medical application in genetics
VMWare production

We will expand this list the next months.

Reinoud

Re: Most unusual application of dedupe?

rkaramchedu1 — Mon, 10 Mar 2008 01:27:48 GMT

Well

Having supported the life sciences community for a few years, when I first heard of de-dupe, I wanted to see if we can use de-dupe algorithms to seek out the commonalities between multiple genomic data sets between organisms, species etc...However, since NetApp de-dupe works on a WAFL block, compares are restricted to 4KB. If that would be configurable, the resulting scenarios would be interesting at best..

that would be a good test for netapp admins currently supporting such data sets.

Re: Most unusual application of dedupe?

lfreeman — Mon, 10 Mar 2008 15:37:02 GMT

Hi reinoud7 - Can you give us an idea of the space savings (%) you are seeing on each of the applications you are deduping? And yes I'd say that the shares with your software source code and the genetics data are "unusual" - good stuff!

Re: Most unusual application of dedupe?

lfreeman — Mon, 10 Mar 2008 15:40:07 GMT

Hi rkaramchedu1, true we are bound by the 4K WAFL blocks. Did you see any saving at all on your genomic data sets?

Re: Most unusual application of dedupe?

reinoud7 — Mon, 10 Mar 2008 16:22:15 GMT

Of course, no problem:

dumps of VMWare: today, it's just a VCB kind of backup: 49 % of savings
dumps of SQL and Sybase : 66 - 68 % of savings (7 full dumps of the same database)
our share with the sources of all our software : only 28 %
medical application in genetics : is till testing, but here we only have a saving 8 %
VMWare production : still in test, more details later but at least 50%

I was forgotten this one: all our invoices, send to our patients (more than one million / year): 52 % (this are pdf-files)

Greetings, Reinoud

Re: Most unusual application of dedupe?

__M_Marotti_3892 — Tue, 27 May 2008 10:53:48 GMT

Talking about unusual application, is there anyone that have been tested SAS institute applications data using dedupe? I want to know the savings.

Thanks,

Re: Most unusual application of dedupe?

lfreeman — Wed, 11 Jun 2008 20:50:55 GMT

Hi M_Marotti - I guess by the sound of the crickets in the background no one has tested SAS Institute data. In a case like this, I've seen people take two approaches:

1) Run the Space Savings Estimation Tool against a sample dataset. This tool will simulate dedupe and is available from your NetApp or authorized VAR SE.

2) SnapMirror a copy of the SAS data to a test/dev volume and run dedupe against that volume

Either of those approaches will help give you an idea of the space savings you'll see.

Hope that helps...

Dr Dedupe

Re: Most unusual application of dedupe?

friea — Thu, 19 Jun 2008 17:57:33 GMT

Hey there ... just noticed you have 99 posts!! Congrats at nearly being into 3 digits ...