topic Re: How can I accurately measure my storage usage? in EF & E-Series, SANtricity, and Related Plug-ins

How can I accurately measure my storage usage?

Hidetoshi_Satou — Tue, 22 Oct 2024 01:03:37 GMT

Customers who want to know how much storage is being used (total allocated capacity across pools or volume groups) should refer to usedSpace in the information retrieved by the Web Services API (ReastAPI).

(Python example)
uri = "https://{0}:{1}{2}{3}/storage-pools/{4}".format(
self.ip_addr,
self.port,
'/devmgr/v2/storage-systems/',
system_id,
uuid)
result = requests.get(
uri,
auth=(self.user, self.passwd),
verify=False,
timeout=20.0
)
used_capacity = result['usedSpace']

The customer informs us that the usedSpace value fluctuates slightly even though the volume is not being extended, created, or written to storage.
(Question 1)
Is there any calculation for usedSpace above other than the amount of storage allocated by the customer in WebGUI?
For example, does it include metadata or snapshot copy data?
(Question 2)
If the above information includes metadata and snapshot information, what information would you look for to get only the amount that the customer has explicitly assigned that does not include it?

Re: How can I accurately measure my storage usage?

elementx — Tue, 22 Oct 2024 12:48:09 GMT

Why don't they look at the capacity value in

curl -X GET "https://1.2.3.4:8443/devmgr/v2/storage-systems/1/volumes" -H "accept: application/json"

If they care about how much they provisioned in terms of volumes, and now abut other fluctuations there may be in pools, snapshot reserves, etc., then volumes is better to look at than pools/groups.

I haven't looked at the output of volumes over time - I assume that does not fluctuate without specific changes to volume sizes (e.g. expansion) by SANtricity admin...

Re: How can I accurately measure my storage usage?

ahmadm — Tue, 22 Oct 2024 17:36:25 GMT

I would add that E-Series does not keep track of actual used blocks on the allocated volume capacity for thick provisioned volumes.

Only if the volumes are thin provisioned (available only on Disk Pools), then the actual consumed capacity can be determined from E-Series end.

Snapshot and other form of repository storage used (i.e Mirroring) is not part of the parent volume allocated capacity. However, it is part of the used capacity at the volume group or pool level.

Re: How can I accurately measure my storage usage?

elementx — Wed, 23 Oct 2024 09:56:13 GMT

> The customer informs us that the usedSpace value fluctuates slightly even though the volume is not being extended, created, or written to storage

What do they have:

- thin volumes on a DDP, or

- thick volumes on a DDP, or

- thick volumes on regular volume groups?

If they look at volume sizes in /devmgr/v2/storage-systems/1/volumes, do those sizes also fluctuate?

Re: How can I accurately measure my storage usage?

Hidetoshi_Satou — Fri, 25 Oct 2024 10:44:27 GMT

Thank you for your reply.

The reason I didn't refer to the size of individual volumes is that I thought it would be easier for a program to implement a single API command to get the usage of multiple volumes in a pool or volume group, rather than counting them individually.

I thought usedSpace was the sum of the sizes of the individual volumes.

The reason for this question is that I wanted to know if UsedSpace includes snapshot reserve space and other variations.(Question 1)

Looking at the sizes of individual volumes and adding them together is an improvement.

Re: How can I accurately measure my storage usage?

Hidetoshi_Satou — Fri, 25 Oct 2024 10:48:14 GMT

Thank you for your reply.

For thickly provisioned volumes, we expected to return the capacity of the volume reserved by the user, not the actual number of blocks used.
"it is part of the used capacity at the volume group or pool level."
I think this comment will answer Question 1 which I wanted to know.

Re: How can I accurately measure my storage usage?

Hidetoshi_Satou — Fri, 25 Oct 2024 10:51:43 GMT

they have thick volumes on a DDP, or thick volumes on regular volume groups.

From what I've heard, it's happening in thick volumes calculations (I don't know if it's DDP or volume groups).

Re: How can I accurately measure my storage usage?

elementx — Fri, 25 Oct 2024 11:12:31 GMT

> I wanted to know if UsedSpace includes snapshot reserve space and other variations.(Question 1)

It's tiresome to figure that stuff out. I created some notes here.

https://scaleoutsean.github.io/2023/10/05/snapshots-and-consistency-groups-with-netapp-e-series.html#appendix-c---repository-utilization

Once I realized how complicated it is, I gave up on trying to do more (such as, include reserve calculation of combined totals (i.e. SUM(volumes + reserves)).

> The reason I didn't refer to the size of individual volumes is that I thought it would be easier for a program to implement a single API command to get the usage of multiple volumes

If they watch that regularly, they could automate and run a subset of this from the CLI.

Then just run a simple SUM to add all volume sizes up.

https://github.com/scaleoutsean/eseries-perf-analyzer/blob/master/collector/collector/collector.py

If reserve volumes were easier to work with and understand I would have included them in some "grand total" metrics, but it didn't seem that way...

If you need any help with the above, you can ask your account team to reach out to me offline.

Re: How can I accurately measure my storage usage?

ahmadm — Fri, 25 Oct 2024 15:40:17 GMT

That's correct.

You cannot determine the actual user utilized space on thick provisioned volumes from the E-Series statistics. This would have to be captured from the filesystem level or whichever application is used on the host end.

Re: How can I accurately measure my storage usage?

Hidetoshi_Satou — Tue, 29 Oct 2024 10:44:05 GMT

Thank you for the notes and sample scripts.
I think the script will answer Question2.
That's all I want to know. Thank you.

Re: How can I accurately measure my storage usage?

elementx — Wed, 27 Aug 2025 17:42:33 GMT

I'll be looking into this again as I am now about to add volume info (not just volume performance) to EPA 4, will post findings here if I find something unusual.

Re: How can I accurately measure my storage usage?

elementx — Fri, 15 May 2026 14:46:03 GMT

Regarding this from the very top:

> The customer informs us that the usedSpace value fluctuates slightly even though the volume is not being extended, created, or written to storage.

I think I know what this is/was - it's the fact that DDP allocates capacity in large chunks, so depending on how small the volume is, you may or may not notice something odd. Generally speaking, it's more obvious below 100 GiB.

See an example in the screenshots in

https://scaleoutsean.github.io/2026/04/13/elasticsearch-eck-kuberntees-netapp-santricity-csi.html#deploy-elasticsearch-9-with-eck

1) I round up to 4GiB on DDP, since it may be allocated anyway

2) I recommend classic RAID (where this doesn't happen) for use cases with many sub-32 GiB volumes