St.node util on some nodes

[sanbre]

Hi,

we recently extended our st.grid env. with 4 nodes per site (8 new nodes total, whole grid now 16 nodes).

What we see in our monitoring that it fills up until 99.88% on 3 nodes, now that we can't handle in our alerting/monitoring.

Is there any way we can spread the load of these full nodes to the other nodes in the cluster or is there a way to set a threshold on the nodes that it fills up to (let's say)90% and no more?

Nodes are virtual (ESXi), not appliances.

[Ontapforrum]

Some pointers - Planning a StorageGRID expansion in the following pdf.  In addition, it may be worth raising a ticket and get more help around this.

Following section talks about rebalancing: (Page-9) 
https://library.netapp.com/ecm/ecm_download_file/ECMLP2873547

What is EC rebalancing?
EC rebalancing is a StorageGRID procedure that might be required after a Storage Node
expansion. The procedure is run as a command-line script from the primary Admin Node. When
you run the EC rebalance procedure, StorageGRID redistributes erasure-coded fragments among
the existing and the newly expanded Storage Nodes at a site

https://kb.netapp.com/Advice_and_Troubleshooting/Hybrid_Cloud_Infrastructure/StorageGRID/How_to_read_the_StorageGRID_EC_rebalance_output

Docs: 
https://mysupport.netapp.com/documentation/docweb/index.html?productID=63606&language=en-US

[Chris_Jones]

I'd definitely recommend upgrading to StorageGRID 11.6.0.4 and then running the EC Rebalance. You get some nice graphs in the Support > Metric > EC Overview section that give you progress status and estimated completion time details.

[aronk]

If you are using Erasure coding then you can make use of the EC rebalancing as mentioned above. If you are only storing replicas, then it is a bit more difficult to rebalance the stored objects.  You can override the read-only watermarks under Configuration/Storage Options. you can also adjust the alert thresholds in Alert/Rules, but as long as there are sufficient nodes with free capacity to to meet the ILM rules, there is no need to do anything about the older full nodes.