This isn't more of a question any more but my experiences with having high metadata consumption on some of the nodes in a multi country stretched NetApp Storagegrid installation using just SG5812 appliances that come with 64GB RAM of which I believe 61GB is actual usable. This means the metadata limit maxs at 3TB per node in total. I am supposed to use only 1.32TB of that capacity according to NetApp recommendations.
Now if there is an imbalance in the total number of nodes in these two countries, like if one country has 6 nodes across 3 sites and the other country has 3 nodes per site in a 3 site solution and the grid is stretched, then you are bound to see high metadata usage is bound to grow in the sites with just 3 nodes.
There is a good documentation that i found as part of my digging to understand how things work on the storagegrid cassandra level. This is my limitation since I wasn't fully versed in this part.
(https://www.netapp.com/video/z0pro187-d8/best-practices-and-advice-for-designing-a-storagegrid-deployment-1351-2/#:~:text=This means that the grid,sto...
So I added virtual metadata-only nodes and as part of the grid expansion the metadata got distributed over to the new nodes, but the compaction jobs didn't take care of reducing the unwanted content on the metadata layer on the storagegrid appliance nodes in all the 3 node sites.
The compaction jobs kept looping over and over and I triggered some manually on nodes that had none running on them. But none of this helped. So, I kept reading into generic cassandra documentation and also storagegrid advanced training documentation where there was a mention of the "nodetool cleanup" command. In this case, the nodes were unable to cleanup what was supposed to be cleaned up even after rebalance.
I chose to go down this path after running the "nodetool status" command and found how there was an extreme imbalance in the Load value on the nodes where clean up was supposed to happen even though the ownership is around 70% odd which it was supposed to be. So, the thought was this is definitely things that needed clean up. So I ran the nodetool cleanup one by one and this brought down the metadata consumption on the existing physical appliances in the 3 node site after addition of metadata. This according to support was not a recommended operation but these appliances seem to be so under powered and CPU busy most of the time and with little RAM not giving the possibility for increasing the metadata capacity.
I have noticed that most of the CPU usage is high and it happens due to cassandra read operations and they seem to contribute to some network retransmissions too.
Another thing i noticed is the kind of data being stored on these appliances are important. If they are very large small objects that create a large load of metadata that will make you reach this problem earlier than you should.
So the solution is to try and keep all of your sites at the same or close by node count so the metadata distribution can happen evenly and catch high metadata consumers and see if they can be reconfigured to a better high write size.
I write this message here since i had a lot of trouble navigating the issue and not article was found help me navigate this situation.
