Dedup only saving 5% on 750gb of windows VM's

[wally_kroeker]

I currently have two ESX 3.5 hosts connected to a 2050 running 7.3.1.1 through iscsi.  I recently had to create a brand new lun and migrate all of my VM's to it becasue the vmfs file system became corrupted.  On my old lun dedup was saving me 65% of my space but on the new one I am only saving 5%.  It is a 1 TB volume with space settings set to volume, fractional space reservation set to 0% and snapshot reserve set to 0%.  On the Lun space reservation is turned off.  I have set dedup to run between 3 and 6 am everyday and it now seems to take 6 min to run every night.  I can't figure out why this would have changed.  Any help would be apreciated.

[keitha]

You are correct, the VM does need to be shutoff for mbralign. Reread my post, I might have said it in an odd way but I did say the VM has to be off for mbralign.

Keith

[eric_barlier]

Radek,

Interesting thought, it certainly warrants investigation. Can anyone from NetApp engineering or other shed any light on this please?

If I was a betting man (never been to Vegas, its not my gig) I d put money on bad MBR align issue not having impact on dedupe savings.

I d still like to know for sure.

Cheers,

Eric

Message was edited by: eric barlier

[renault]

All,

Reading this topic I think two possible reasons of poor dedup performance

- The directories of your VMs are compressed. Compressed data cannot be deduplicated

- Use of crypt engine like decru. Crypted data cannot be deduplicated.

My VM are not block aligned and we get 50% of dedup perf.

You may also zeroes the free space of your VMs

And if you have only 2 ESX host, you shoud try NFS storage. But it's an other subject.

Just my 2 cents

ML

[keitha]

I have been working with Wally on this issue and am stumped. Here is an update and a couple of things that still baffle me. First we have checked and rechecked the LUN and volume reservation settings. They are correct. Second although I agree alignment is likely an issue, these same VMs were deduping at nearly 70% beore Wally copied them to this new LUN. Same VMs and the VMware copy process shouldn't have affected the dedupe rate.

Now the really odd bit. As we deploy new VMs or align the old once they seem to dedupe at the right ratio, last I spoke to Wally the ratio was 15% and climbing. Perhaps the copy process did mess something up in the alignment process. Anyhow it does seem to be fixing itself as we align the VMs correctly using MBRALIGN.  Thanks to everyone for the suggestions.

Keith

[eric_barlier]

Hi keith,

I reckon how you did the copy will be a factor here. Say if you were to use NDMPCOPY that will nicely restripe your block layout. This is good

if your volume is fragremented, NDMPCOPY a volume is a known workaround for fixing defrag issues. however as the blocks are re-striped I think

it will undo what dedupe did as the block layout will have changed. Does that makes sense?

Interesting topic still as Mark-Laurent DeLaruelles statement opposes what you have observed.

Cheers,

Eric

[wally_kroeker]

I think I have solved my problem when all of my vm's most of which are 2003 servers where out of alignment.  when I did the alignment they started deduping.  that being said I think the reason this fixed it is becasue dedup is esentially seeing the alignment as new data.  I think the reason this all started is becasue I turned on dedup with the windows app "system manager"  the default action of this app is to not scan old data but just new.  Keith Asen from netapp said that if I had kicked off dedup with sis start -s then it would have worked from the start.  THanks for all the help everyone.

[radek_kubka]

To make our discussion complete...

It looks like we were (sort of) reinventing the wheel - this is what I just found in TR-3749 on page 69:

12.4 THE IMPACT OF PARTITION MISALIGNMENT
[...]
In addition to the negative performance impact storage savings with NetApp data deduplication will be
negatively impacted, reducing the total amount of storage savings.

Nice one

Regards,

Radek