Defeating NetApp "Right Sizing" for more Usable Capacity

[bepresseditor]

I'm interested in learning more about NetApp "Right Sizing" of drives, with the hope of recovering more usable capacity. For example let's take a brand-new FAS2020 with 1TB Drives:

The drives report out as X298_SMOOS01TSSS, which maps to IBM part number 45E2141.  The actual drive appears to be a Seagate Barracuda  ES.2 Model ST31000640SS  (or ST31000640SS for SAS).  The vendor specifications for that drive ( Barracuda ES.2 SATA Product Manual ) show "guaranteed sectors" of 1,953,525,168. Reformat that from 512 bytes/sector to 520 and you should have 1,923,470,934 raw usable sectors, at a very minimum, accounting for all drive overhead and all drive remapping spares.

Yet the actual filer reports a significant NetApp tax:

  RAID Disk Device  HA SHELF BAY CHAN Type  RPM  Used (MB/blks)    Phys (MB/blks)
  --------- ------  -- ----- --- ---- ----  ---- --------------    --------------
  dparity   0c.00.1 0c   0   1   SA:B SATA  7200 847555/1735794176 847884/1736466816
  parity    0c.00.4 0c   0   4   SA:B SATA  7200 847555/1735794176 847884/1736466816

Which is line with the loss predicted at: http://communities.netapp.com/servlet/JiveServlet/download/21878-11275/Right-sized%20disks.pdf ("Right-sized Disk Capacity").

So we lost basically 10% before we even began the loud bickering between storage vendors.

Now the official reason for all this is found direct from NetApp:

"Disk drives from different manufacturers may differ slightly in size even though they belong to the same size category. Right sizing ensures that disks are compatible regardless of manufacturer. Data ONTAP right sizes disks to compensate for different manufacturers producing different raw-sized disks."

But this seems like nonsense for several reasons.  First, every vendor will produce 1TB drives with at least 1,000,000,000 raw usable bytes, because otherwise they can't sell them as 1TB drives. So we ought to get at least that much.  Second, NetApp should have enough market power to insist that each Barracuda sent to NetApp have a certain minimum number of actual sectors (Seagate can sell the slightly smaller ones as desktop drives via some bottom feeding low price vendor).  We're paying enough premium for the drives, someone else should adsorb the cost of drive size varriance.  And Third, I fully expect to be able to get Barracuda drives on eBay well into the next millenium.  I'm not swapping out the rest of my drives, I'll always buy a similar one, so I don't care about supporting multiple vendors.

So I'm curious if there is a way to trick OnTAP into using more sectors.  On a 12 drive filer with RAID-DP, that's well over 1 Terrabyte of missing usable storage space.

[amiller_1]

The short answer is....no.

For what it's worth, while you can get the exact same drive model elsewhere (i.e. eBay for the next decade), drives from NetApp do have NetApp custom firmware....so a drive out of a Seagate box won't work.

[bepresseditor]

I rather was looking for the long answer.

That custom NetApp firmware benefits me how?  By reducing usable capacity 11.3%?  By making it less confusing to purchase as I have just one option?  Really, I don't begrudge something that's an expensive benefit, but as far as I can tell "right sizing" is all downsides for the customer.  Am I missing a benefit?

[mheimberg]

IMHO the arguments of bepress-editor are worth a bit a longer answer...

Another "waste" of space IMHO nowadays is the fix 10% WAFL reserve - in times of 36G or 72G 10% might have been appropriate....but nowadays with 2TB or 600G drives this is a lot of space for ....what...?? Wouldn't it be possible to fix the WAFL reserve to just 20G or so?

regards

Markus

[radek_kubka]

Hi all,

First of all, a pretty fundamental question:

Which reseller actually talks to the customer about raw capacity, not usable capacity? If they do, this is a very bad practice almost always leading to a customer being disappointed.

I *always* talk about usable capacity, explaining thoroughly how it has been calculated & what assumptions have been made.

If we look at this from that perspective, 'issues' described in this thread simply go away - one has been told how much *usable* capacity they will get & one is paying for this capacity, not for a marketing myth of raw capacity.

Re calculations:

1) Disk manufacturers actually do lie - to them 1GB equals (as per Seagate datasheet) "one billion bytes", whilst if you do proper base 2 arithmetics (1024*1024*1024) 1GB in fact is 1,073,741,824 bytes. So at the very beginning your drive marketed as 1TB is in fact ~931GB.

2) So called right-sizing is due to the fact how WAFL formats the drive - it uses block checksums disks & for SATA drives it means there are 8x 512 bytes sectors holding actual data, followed by 64 byte checksum, followed by unused space of 448 bytes (long story short, that white space is to keep the same checksum size for FC & SATA drives). So the right-sized capacity of the "1TB" drive is ~828GB.

3) WAFL reserve - "approximately 10% of space in  the aggregate for aggregate level metadata, and for performance reasons". You may argue with the latter that it should be adjusted because disks got bigger, but you still need the same proportion of space for the former as long as WAFL uses 4kB blocks (more blocks on the disks, more metadata).

Good NOW article:

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb34044

But, again - normally I am not bothered about all this numbers, as I just look at the end result, i.e. usable capacity (there are NetApp tools for calculating this) & equally my customers do not need to worry, as I *never* sell them raw capacity.

Hope it helps.

Regards,
Radek

[mheimberg]

Hi Radek

First of all, a pretty fundamental question:
Which reseller actually talks to the customer about raw capacity, not usable capacity? If they do, this is a very bad practice almost always leading to a customer being disappointed.

eg. NetApp themselves by selling shelves "DS14 14TB" or "DS14 7TB" with 1TB SATA resp. 750GB ...some months ago I had the pleasure to install a system that NetApp has sold directly: the customer calculated for 40 or something TB and was quite surprised when he ended up with only 25TB usable...so obviously no one of the NetApp sales explained him the calculation - and they didn't ask, what is bad either....

Me too, I try to tell the customer the truth and explain the maths. Making quotes I always ask "how much do u need" (+growth + delta + snap retention what is often not so easy to get...)

If we look at this from that perspective, 'issues' described in this thread simply go away - one has been told how much *usable* capacity they will get & one is paying for this capacity, not for a marketing myth of raw capacity.

But sometime the "evil competitors" don't play fair...but then we get the chance to be recognized as "trusted consultant", don't we?

3) WAFL reserve - "approximately 10% of space in  the aggregate for aggregate level metadata, and for performance reasons". You may argue with the latter that it should be adjusted because disks got bigger, but you still need the same proportion of space for the former as long as WAFL uses 4kB blocks (more blocks on the disks, more metadata).

hrm...ok, more blocks = more metadata....what is "metadata" in this context? Well, nevermind: I know this goes deep into WAFL and I am not sure that I would understand it even if you could explain it to me 🙂

regards

Mark

[bepresseditor]

Ok, well that answers part of the missing data.  The Segate Barracuda  ES.2 Model ST31000640SS  "guaranteed sectors" of 1,953,525,168 times the 8/9 ZCS checksum tax is:

1,953,525,168 * (8 / 9) = 1,736,466,816

Which is exactly the number of raw blocks reported by the FAS2020.  So apparently RightSizing has far less to do with drive vendor differences, and more to do with checksums. Apparently NetApp does not have enough market power to get Segate to format these drives with BCS 520 byte sectors, so we loose one block out of every 8.  This despite the fact Segate maintains custom firmware for the NetApp model.

Got it.  Is that "wasted" 488 bytes ever put to good use, such as for undelete or cross-check metadata?

[mheimberg]

AFAIK this has nothing to do with "market power" but rather with historical, technical reasons: FC disks have 520B blocksize with 512B for data and 8B for chechsum, so checksum is integrated into the block. SATA/SAS in contrast don't. So if you want the same reliability you have to trick as explained with taking some block (8x512B) and add the chechsum additionally, thus loosing some bytes....

regs

Mark

[DaleRB]

Correction: FC and SAS disks have 520B blocksize with 512B for data and 8B for chechsum, so checksum is integrated into the block. SATA in contrast doesn't.

FC and SAS drives use 520-byte sectors. FC and SAS drives are formatted using the block checksum scheme (BCS) method. Each 4KB WAFL® (Write Anywhere File Layout) block is made up of eight sectors. The last eight bytes of each sector are reserved to store the block checksum. This means that each 520-byte sector is effectively 512 bytes of usable capacity. In doing the math, you end up with 512 bytes x 8 = 4,096 bytes, or a 4kB WAFL block.
SATA drives start with 512-byte sectors. In order to store block checksum information, Data ONTAP uses the BCS 8/9 checksum method when formatting the drive. This means that each 4kB WAFL block is made up of nine sectors, eight that are used to store data and the ninth sector used to store the checksum (8/9).

See: http://www.netapp.com/us/media/tr-3838.pdf

[DaleRB]

FYI, with Data ONTAP 8.1.1 NetApp introduced a new “Advanced Zone Checksum” style (“AZCS”) for native 4KB formatted HDDs and for V-Series LUNs. From a capacity perspective it has a cost of of 1 – 63/64 or 1.56% of the HDD capacity vs. the previous BCS checksum style emulation on 512bps drives that was 1 – 8/9 = 11.1% of the HDD capacity. The 4KB native disks are of type MSATA found in the DS4486 shelf in 3TB and 4TB capacities and are shipping today.

I try to always distinguish between binary and decimal using the ANSI standards to avoid confusion.   i.e. 1GB=1,000,000,000 bytes;  1 GiB=1,073,741,824 BYTES.

[amiller_1]

I much appreciate the longer answer from Radek...I wasn't trying to be rude...just something I've pushed on at various times internally and have been told "that's the way it is". When I'm talking with customers, I just say some of it is for "WAFL-ey" goodness (after working through the Base 10 vs. Base 2, spare disks, WAFL overhead, etc.).

Having said that, it's really more a pricing thing...always, always talk usable space and it's up to sales to get the pricing where it needs to be.

[mortenmadsen]

---

@radek_kubka wrote:

"1) Disk manufacturers actually do lie - to them 1GB equals (as per Seagate datasheet) "one billion bytes", whilst if you do proper base 2 arithmetics (1024*1024*1024) 1GB in fact is 1,073,741,824 bytes. So at the very beginning your drive marketed as 1TB is in fact ~931GB."

Well that is not a lie, the "G" in GB is the giga prefix from the SI units and means 1,000,000,000 (https://en.wikipedia.org/wiki/International_System_of_Units) Why any computer guy thought it would be a good idea to use incorrect prefixes only the gods will know. They should have used/invented the binary prefix instead: https://en.wikipedia.org/wiki/Binary_prefix

[bepresseditor]

Ah, I did finally manage to get a generic off-the-shelf $100 Segate 1TB drive to work in the FAS2020.  Took some doing, is possible 🙂

[mheimberg]

And of course you still expect NetApp to exchange your failed drives within 4 hours....?

Serious: from an engineering point of view it is very fascinating what you can do (reminds me of those guys who are trying to break into Playstation or Wii etc) - but what's the benefit in real world? Neither do most customers (or even partner) have the knowledge, will or time to do such tricks nor will they take the loss of guarantee into account.

And you did it in a cheesy low-budget 2020 to save yet some other bucks....

Nonetheless I agree with you, and I am still missing some exprehensive explanations from NetApp, why we have to loose so much space. Especially when disk pricing is not far away from robbery...

Mark

[eric_barlier]

I thought these drives were downsized out of disk factory, not by NTAP and that its the same across all storage vendors, is that not the case?

A bit like as an end user in a shop I ve NEVER been able to get the disk size I was sold. Every disk I ever bought was down sized no matter what.

I dont see this as a NTAP specific issue, rather an industry standard.

I assume you are out of warranty? Well done on getting it to work though.

Eric