reallocate TR?

[danpancamo]

After fighting with our DBAs for months now about IO performance, we finally narrowed down a growing performance issue to disk fragmentation.   We accidentally discovered that a copy of a database was 2x faster to backup than the original copy which lead us to the conclusion that we did actually have an issue with IO. 

We ran a reallocate measure which resulted in a threshold of 3.   I'm not sure exactly how this is calculated, but obviously the result is misleading. The results should have been 10 IMHO.

We ran reallocate start -f -p on the volume and it immediately reduced the backup time in half.   Disk util, disk caching, latency all were significantly better after the reallocate completed.

It appears that a the Sybase full reindex basically tries to optimize by re-writing data...   This process somehow causes the disk fragmentation.

I've only been able to find limited information on the reallocate command, however with this significant performance increase, there should be a whitepaper on the subject that includes the effects of reallocate on Cache, PAM, Dedup, Vmware, Databases, Exchange, etc...

Is there a TR document in the works on reallocate?  If not, someone should start one.

[erick_moore]

Well you can enable read_realloc on volumes after 7.3.1.  I agree that it would work better if we didn't have to manage this at all.  Let the system realloc in the background while idle.  I fully support a wizard when adding new disk to an aggregate that ask me if I want to reallocate all my volumes for that aggregate in the background.

[jasonczerak]

yeah, but that's a real time CPU and IO hit if it has to reallocate after a read.  And isn't that a bad idea when using de-dup?  As far as the wizard idea, so long as the reallocate DOES NOT imact latency, sure.... 

There needs to be a chart of what "features" work well togehter and which ones shouldn't.  like read_reallocate and extents and dedup + reallocate = bad or something along them lines.

[erick_moore]

Dedup blocks can't get reallocated.  They are stuck where they are once deduped.  We haven't seen any performance hit with read_realloc since it only reallocates the portions of the volume that are being asked to be read sequentially.  In fact, I don't even know if read_realloc actually reallocates blocks or if it just updates the metadata for the readset to include the blocks that it wouldn't normally get on that read request.

[jasonczerak]

Intresting. that would make sense I suppose.

Stil tho, it would be nice to have a nice chart out side of default options.

Not everyone has a "test" netapp.  I've been lucky with 2 3160 clusters for the past 6 months to do some major expermients with. but that luxary is going away soon for me, back to experimenting on production hardware 😉

[radek_kubka]

Dedup blocks can't get reallocated.  They are stuck where they are once deduped. 

Interesting stuff.

Are you sure this is also the case when using reallocate -p (physical) option?

Regards,
Radek

[erick_moore]

I know for a fact it doesn't work with read_realloc per TR-3505, and I would assume the same applies to standard reallocation.

[aborzenkov]

Even if they were reallocated, you can’t please ‘em all. You have a block that is shared by dozen of files. (Logically) adjacent block is shared by dozen different files. You have no way to reallocate them both and keep all files that share these blocks sequentially laid out.

So it is not that they are really set in stone – it is more that shuffling de-duplicated blocks around is not going to improve total picture much; de-duplicating one data set will inevitably fragment another.

---

With best regards

Andrey Borzenkov

Senior system engineer

Service operations

[aborzenkov]

Cleaning up disk I found tr3599 which is titled "REALLOCATE BEST PRACTICES GUIDE " Now it is marked confidential and I am not able to find it on fieldportal (which does not mean anything, it is near to impossible to find anything on purpose there). It is dated 2007 so I guess I have it from old partner portal. It does not contain anything excitingly new that was not already covered in various discussions; the main value is, it represents official NetApp position about use of reallocation.

I'd love to see it updated to include new physical/aggregate reallocation. Anyone knows if newer version exists or why it was removed?

[danpancamo]

running: Release 7.3.2P2:

We continue to see 50%  (yes 50%) or more gains after some reallocate runs. Mainly heavy sequental reads and mainly on large database files that get updated frequently...

TIP:  We have discovered that if you are using a NFS volume you can run reallocate on individual files instead of the whole volume which can take the reallocate time down from hours to minutes.    However you need to create a schedule for each file.

TIP: We have seen reallocate measure results over 20, so 10 must not be the max..

new stuff....

I'm back posting today to try and understand the effects of running reallocate on our VDI VMWARE volumes which are deduped.   reallocate measure is reporting a 6 with a hotspot 19 with a recommendation to run reallocate.

From reading the above posts, someone stated that deduped blocks cant be reallocated, so does that mean that on a volume with multiple duplicate VMs, reallocate wont do anything?   

We are seeing a volume latency issue  (2 to 60ms) when deduplication and virus scans run at the same time.    I initially thought that reallocating should speed on the virus scans, however if dedup blocks are not reallocated this may do nothing??

what does hotspot 19 mean?

Again I vote for a TR on this!

Dan Pancamo

[radek_kubka]

Hi Dan,

I'm loving this thread more & more! Many people at NetApp are still in a constant state of denial that fragmentation even exists

Re dedupe & reallocation - what Andrey wrote above makes sense to me:

Even if they were reallocated, you can’t please ‘em all. You have a  block that is shared by dozen of files. (Logically) adjacent block is  shared by dozen different files. You have no way to reallocate them both  and keep all files that share these blocks sequentially laid out.

Having said this, if you can test it in practice, that would be awesome (is reallocate a silver bullet for every problem? )

Regards,

Radek

[aborzenkov]

We continue to see 50%  (yes 50%) or more gains after some reallocate  runs. Mainly heavy sequental reads and mainly on large database files  that get updated frequently...

I have seen 200% on backup (dump) of volume with database (50MB/s => 200 MB/s). This was test run (dump to null); real life is limited by other factors still it cut down backup time in half.

We have seen reallocate measure results over 20, so 10 must not be the max..

Max is 32. I have seen over 20 on the system I mentioned as well.

reallocate measure is reporting a 6 with a hotspot 19

Could you show exact message and where do you get it? I do not remember having seen it.

[rnugent2068]

The search for reallocation info continues . . .

aborzenkov wrote:

We have seen reallocate measure results over 20, so 10 must not be the max..

Max is 32. I have seen over 20 on the system I mentioned as well.

reallocate measure is reporting a 6 with a hotspot 19

Could you show exact message and where do you get it? I do not remember having seen it.

I don't think 32 is the top of the scale. I just got this from a reallocate measure:

[server1: wafl.reallocate.check.highAdvise:info]: Allocation check on '/vol/TEST/test.lun' is 36, hotspot 0 (threshold 4),

consider running reallocate.

Does anyone know where I can find a definitive resource for reallocation info?  NetApp engineers - are you there ? 

(Referred to this post from my original question: http://communities.netapp.com/message/47200#47200 )

Thanks

[aborzenkov]

That’s quite possible. In the past target allocation size was 64K (resulting in 16), then it was increased to 128K (giving 32); I can well believe today it is increased even more. 256K does not sound like unreasonable.

Anyone knows counters related to this (i.e. how many full extents were written)?

[aborzenkov]

TR-3929 Reallocate Best Practices Guide

https://fieldportal.netapp.com/viewcontent.asp?qv=1&docid=33904

Not that it contains anything that was not already beaten to death here ...

[radek_kubka]

Wow, I can't believe my eyes - it really happened (at last)!

The doc itself is rather brief I would say & arguably doesn't cover many corner cases, which typically are the most confusing ones.

That said, it is way better than nothing!

(NB - not available via http://www.netapp.com/us/library/, so the subject is still "sensitive" I reckon)

Regards,
Radek

[igor]

In so many words, we've 2 reallocate approaches:

     Traditional reallocate - does the job, introduces snapshot space impact, SnapMirror impact

     Physical reallocate - does the job, introduces snapshot read impact, no SnapMirror impact

And then, there's also Read reallocate option for volumes (didn't know about this one) - which follows the same principles, but uses regular read workload for optimization rather than scanning mechanism, and is apparently recommended to work along side regular reallocate processes. Interesting... "optimal layout of frequently accessed data".

I wonder when exactly does Read reallocate take place? Because, if it's performance demanding I wouldn't like it kick in just anytime. Also, am I right to assume it's performance impact will reduce over time, after running a few passes under the same work load and slow build-up of data?

[aborzenkov]

I was always uneasy about read reallocation. It appears to take place at exactly wrong time – after we already paid penalty of non-sequential read ☺ Depending on environment, next time we need to read data it may have been fragmented again … so it apparently makes sense only in highly static environment.

I wish this TR explained how can we estimate (or better – get real counters) of how effective read reallocation was. Something about how often reallocated data was read subsequently before being fragmented again.

[jakub_wartak]

Hi,

i've started playing with this in a more repeatable/scientific way. Results are on my blog and more are about to come:

1) WAFL performance VS sequential reads: part I, FC LUN performance from AIX vs FlexVol utilization <-- intro, contains descripton of the env used, i know it is not ideal but i don't have anything better currently without requiring Change Controls. http://jakub.wartak.pl/blog/?p=316

2) WAFL performance VS sequential reads: part II, FC LUN defragmentation <- a nice show case for "reallocate start -f -p".  http://jakub.wartak.pl/blog/?p=330

-Jakub.

Message was edited by: Jakub Wartak

[aborzenkov]

Could you remind where your blog is?

[jakub_wartak]

Yeah, i've forgot to include the links - orginal post was edited

[dnewtontmw]

Is anyone else having trouble getting to this doc?

If I click on the TR link above, I get error messages about being unable to log in (to the iCentera site, which tries to redirect to the fieldportal URL.)...