How can I accurately measure my storage usage?

[Hidetoshi_Satou]

Customers who want to know how much storage is being used (total allocated capacity across pools or volume groups) should refer to usedSpace in the information retrieved by the Web Services API (ReastAPI).

(Python example)
uri = "https://{0}:{1}{2}{3}/storage-pools/{4}".format(
    self.ip_addr,
    self.port,
    '/devmgr/v2/storage-systems/',
    system_id,
    uuid)
result = requests.get(
     uri,
     auth=(self.user, self.passwd),
     verify=False,
     timeout=20.0
)
used_capacity = result['usedSpace']

The customer informs us that the usedSpace value fluctuates slightly even though the volume is not being extended, created, or written to storage.
(Question 1)
Is there any calculation for usedSpace above other than the amount of storage allocated by the customer in WebGUI?
For example, does it include metadata or snapshot copy data?
(Question 2)
If the above information includes metadata and snapshot information, what information would you look for to get only the amount that the customer has explicitly assigned that does not include it?

[elementx]

Why don't they look at the capacity value in

curl -X GET "https://1.2.3.4:8443/devmgr/v2/storage-systems/1/volumes" -H "accept: application/json"

If they care about how much they provisioned in terms of volumes, and now abut other fluctuations there may be in pools, snapshot reserves, etc., then volumes is better to look at than pools/groups.

I haven't looked at the output of volumes over time - I assume that does not fluctuate without specific changes to volume sizes (e.g. expansion) by SANtricity admin...

[Hidetoshi_Satou]

Thank you for your reply.

The reason I didn't refer to the size of individual volumes is that I thought it would be easier for a program to implement a single API command to get the usage of multiple volumes in a pool or volume group, rather than counting them individually.

I thought usedSpace was the sum of the sizes of the individual volumes.

The reason for this question is that I wanted to know if UsedSpace includes snapshot reserve space and other variations.(Question 1)

Looking at the sizes of individual volumes and adding them together is an improvement.

[elementx · Accepted solution]

> I wanted to know if UsedSpace includes snapshot reserve space and other variations.(Question 1)

It's tiresome to figure that stuff out. I created some notes here.

https://scaleoutsean.github.io/2023/10/05/snapshots-and-consistency-groups-with-netapp-e-series.html#appendix-c---repository-utilization

Once I realized how complicated it is, I gave up on trying to do more (such as, include reserve calculation of combined totals (i.e. SUM(volumes + reserves)).

> The reason I didn't refer to the size of individual volumes is that I thought it would be easier for a program to implement a single API command to get the usage of multiple volumes

If they watch that regularly, they could automate and run a subset of this from the CLI.

Then just run a simple SUM to add all volume sizes up.

https://github.com/scaleoutsean/eseries-perf-analyzer/blob/master/collector/collector/collector.py

If reserve volumes were easier to work with and understand I would have included them in some "grand total" metrics, but it didn't seem that way...

If you need any help with the above, you can ask your account team to reach out to me offline.

[Hidetoshi_Satou]

Thank you for the notes and sample scripts.
I think the script will answer Question2.
That's all I want to know. Thank you.

[ahmadm]

I would add that E-Series does not keep track of actual used blocks on the allocated volume capacity for thick provisioned volumes.

Only if the volumes are thin provisioned (available only on Disk Pools), then the actual consumed capacity can be determined from E-Series end. 

Snapshot and other form of repository storage used (i.e Mirroring) is not part of the parent volume allocated capacity. However, it is part of the used capacity at the volume group or pool level.

[Hidetoshi_Satou]

Thank you for your reply.

For thickly provisioned volumes, we expected to return the capacity of the volume reserved by the user, not the actual number of blocks used.
"it is part of the used capacity at the volume group or pool level."
I think this comment will answer Question 1 which I wanted to know.

[ahmadm]

That's correct.

You cannot determine the actual user utilized space on thick provisioned volumes from the E-Series statistics. This would have to be captured from the filesystem level or whichever application is used on the host end.

[elementx]

> The customer informs us that the usedSpace value fluctuates slightly even though the volume is not being extended, created, or written to storage

What do they have:

- thin volumes on a DDP, or

- thick volumes on a DDP, or

- thick volumes on regular volume groups?

If they look at volume sizes in /devmgr/v2/storage-systems/1/volumes, do those sizes also fluctuate?

[Hidetoshi_Satou]

they have thick volumes on a DDP, or thick volumes on regular volume groups.

From what I've heard, it's happening in thick volumes calculations (I don't know if it's DDP or volume groups).