Solved: Re: Freeing up space in an aggregate

csboradba · ‎2011-01-31

I have an aggregate that is %100 full with two volumes in it. I want to reduce the space in one volume and then use that same space to increase the size of the other volume.

However when I reduce the volume size by 15GB it does not release the space back to the aggregate. Still reporting %100 full. Is there something else I need to do to reclaim this space?

aggr show_space outputs the following:

Volume Allocated Used Guarantee
NL_bkp 13801975092 KB 13801975032 KB Volume
PD 613827720 KB 124882836 KB Volume

Aggregate Allocated Used Avail
Total Space 14415802812 KB 13926857868 KB 0 KB
Snap reserve 753608860 KB 12641976 KB 740966884 KB
WAFL reserve 1674686360 KB 1035448 1673650912 KB

Could this be an over-provisioning and I am merely freeing up space which is then taken back by the over-provisioned volume? If so how would I calculate how much space I need to turn on space guarantee?

Thanks

martin_fisher · ‎2011-02-02

the command: "snap reserve -A" and this will tell what if any snap reserves you have set for the aggregates.

snap list -A will show you any snapshots for the aggr's..

the output of df -Ah shows the aggregate size, utilization and percentage used, and free space (if any). eg: FA is 13TB in size, 13TB has been used, zero GB available and the aggregate is 100% utilized.

FA/.snapshot shows information regarding the space reserved for snapshots of your aggregate.

So in the example below it looks like you have aggregate snap reserves set as you 718GB reserved for snapshots. This is approx 5% of your total aggregate size and is a default setting within ONTAP.

Aggregate            total    used      avail capacity
FA                       13TB    13TB     0GB 100%
FA/ .snapshot     718GB 651MB 718GB 0%

The aggregate snapshots are only consuming 651MB of space.

You can resize the reservation for the aggregate snapshots by entering the command:

"snap reserve -A <aggregate name><percentage>"

Say you resize to 4%, the aggregate would then only have 4% of the total aggregate space, for aggregate snapshots.

The space returned should then appear as free space for the aggregate and bring your 100% utilization down.

This could be a temporary measure to provide free space, but if you have something eating all the free disk space and the volumes are over provisioned, the free space will be consumed quickly.

View solution in original post

martin_fisher · ‎2011-01-31

Hi, 100% utilization for any aggregate is not good, you will see plenty of info regarding this in many threads on the communities.

Returning Space to the aggregate from resized volumes can take a few minutes to show up on the statistics/df/aggr show_space commands.

You will not be able to assign any more free space to the volumes, if the utilization is 100%. You would eventually be able to reassign the space to another vol once the system shows it as free. Really you need to get the space utilization down under 65% or at least 85% definitely for an aggregate.

Have you checked how many snapshots your are retaining for the volumes, or the snapshot schedules.

Your vols, have the volume gurantee's switched on. You could remove these so the space is not guaranteed, but then the storage maybe overprovisioned even further!!.

Are you able to delete any of the data held within the shares at all.

what results do you get if you run:

df -Ah

Thanks

csboradba · ‎2011-02-01

Thanks for the reply. I have seen some of the other comments regarding utilization levels and will defintely look at this in the future.

I have left the system for nearly 24 hours now and still can not see the space become available? I have also found that I cannot add the space back to the volume it seems to have just disappeared. I assume because of over-provissioning?

The results of df -Ah

Aggregate            total    used      avail capacity
FA                       13TB    13TB     0GB 100%
FA/ .snapshot     718GB 651MB 718GB 0%

martin_fisher · ‎2011-02-01

Unfortunately it looks like you will have to clear more than 15GB of space, before your utilisation comes down.

If you perform a "snap list <volume name>", e.g: snap list vol0, you will be able to see how many snapshots you have for the vols on your aggregate.

You can delete some of the larger,older snapshots to return some of space to volume as a temporary measure, but it looks like a review of the data held on there is in order.

Reducing the amount of space reserved for aggregate snapshots would also return some space to the system, but if your volumes are over provisioned, then this would be filled pretty quickly.

csboradba · ‎2011-02-02

Thanks

The NL volume does not have snapshots turned on.

This is the output for PD

%/used %/total date name
0% ( 0%) 0% ( 0%) Feb 02 08:00 hourly.0
0% ( 0%) 0% ( 0%) Feb 02 00:00 nightly.0
0% ( 0%) 0% ( 0%) Feb 01 20:00 hourly.1
0% ( 0%) 0% ( 0%) Feb 01 16:00 hourly.2
0% ( 0%) 0% ( 0%) Feb 01 12:00 hourly.3
0% ( 0%) 0% ( 0%) Feb 01 08:00 hourly.4
1% ( 1%) 0% ( 0%) Feb 01 00:00 nightly.1
1% ( 0%) 0% ( 0%) Feb 31 20:00 hourly.5

Would I be right in thinking that because snapshot reserve is set to 0% deleting space from the volume will not free up space because it will be used by the

snapshots?

Is the FA/ .snapshot (df -Ah) a seperate snapshot for the aggregate itself or is it a total of all volume snapshots that are contained in the aggregate.

Can I then use this to calculate the amount of space I would need to free up to get to the stage where addtional freed space would be available to the aggregate

rather than swallowed up by what I am assuming is the snapshots?

martin_fisher · ‎2011-02-02

the command: "snap reserve -A" and this will tell what if any snap reserves you have set for the aggregates.

snap list -A will show you any snapshots for the aggr's..

the output of df -Ah shows the aggregate size, utilization and percentage used, and free space (if any). eg: FA is 13TB in size, 13TB has been used, zero GB available and the aggregate is 100% utilized.

FA/.snapshot shows information regarding the space reserved for snapshots of your aggregate.

So in the example below it looks like you have aggregate snap reserves set as you 718GB reserved for snapshots. This is approx 5% of your total aggregate size and is a default setting within ONTAP.

Aggregate            total    used      avail capacity
FA                       13TB    13TB     0GB 100%
FA/ .snapshot     718GB 651MB 718GB 0%

The aggregate snapshots are only consuming 651MB of space.

You can resize the reservation for the aggregate snapshots by entering the command:

"snap reserve -A <aggregate name><percentage>"

Say you resize to 4%, the aggregate would then only have 4% of the total aggregate space, for aggregate snapshots.

The space returned should then appear as free space for the aggregate and bring your 100% utilization down.

This could be a temporary measure to provide free space, but if you have something eating all the free disk space and the volumes are over provisioned, the free space will be consumed quickly.

chriskranz · ‎2011-02-03

If you have grown your aggregate to 100%, then you may be in other issues such as volumes at over 100%. If a volume grows to over 100%, it won't show you that it's used more storage, but will simply use more storage in the aggregate. This can happen if you have large snapshots and also have fractional reservation enabled. I've seen volumes that occupy an additional 20-30% of their actual allocated size. This is sort of a design feature as free-space is free-space.

However I can't emphasise enough how you are masking a huge problem. Get your aggregate down below 90% before you start trying to extend any existing volumes, you are causing yourself a world of pain. Apart from anything else you are now being hugely innefficient with any write is happening, and that will permanently affect your read performance (until you do something about it). If this is the root aggregate, you are also in trouble of causing yourself system instability.

No file system likes being at 100%, so why do this to core storage?

csboradba · ‎2011-02-03

Thanks both. you have provided me with some really useful information. I have taken a look at some of the data on the NL volume and I am now in a position to shink the volume by 3TB. I am hoping this will be enough to free up space in the aggregate and get the storage to a more 'sane' level.