Root Volume Corrupt

[TMADOCTHOMAS]

The root volume on a node in our simulator bombed out over the weekend for some reason. It says "ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED". After logging in, "Cannot open corrupt replicated database. Automatic recovery attempt has failed or is disabled. Check the event log for details. This node is not fully operational. Contact support personnel for the root volume recovery procedures".  I can't boot to the LOADER prompt. I restored from backup but am still getting the error. Any recommendations? Any idea what happened that I might be able to prevent it from future rebuilds? Thanks!

[cedric_renauld]

I've the same on my simulator when the root vol come to be full

Tho check it you can boot in Maintenance mode, by press the CTRL-C @ boot and chose 5) in boot menu

after check the aggr status, besause you dont have all command available

[TMADOCTHOMAS]

Thank you! Unfortunately, maintenance mode isn't even working. I can press 5 but it never loads to a prompt. I think the cluster is dead 😕 which is unfortunately my 2nd dead simulator cluster in three months. 

I had done a lot of the root vol expansions when first building the cluster, but maybe not enough. Or maybe there's a way to prevent space from filling up. Any suggestions to minimize the chances of this happening again?

[TMADOCTHOMAS]

Regarding my last comment, I should note I've already done the basics like turning off aggregate snapshots, removing the schedule, etc.

[cedric_renauld]

Yes, I've the same issue some time, my only advice is : Snapshot (VMware or other) to preserve a good version of your SIM ...

[TMADOCTHOMAS]

Thanks cedric. Believe it or not I actually did that this time and restored from backup, but the issue still occurred ... must have been the root vol filling up fast for some reason (even though nothing was happening on the sim at the time).

[TMADOCTHOMAS]

Here's a broader question: what's the ideal amount of space to give the root aggregate/volume on the simulator?

[Ontapforrum · Accepted solution]

I personally never had issue with restoring from snapshot (vmware), I have had different snapshots taken at different times (for different PowerShell versions) and every time Simulator crashed I just restored it back. I have also upgraded my Simulator few times as well. I checked my Simulator for the root aggr/vol space, it has 6GB(aggr) and 5GB (vol0) assigned, with approx. 3GB remaining. May be this is a lot but it serves me well.

On this topic, I googled, and found this blog, the blogger suggests '5GB' for vol0. I am guessing, anything between 5-7GB should be resilient enough ?

NetApp simulator: Prevent the root volume from filling up:
https://blog.milla-online.de/netapp-simulator-prevent-the-root-volume-from-filling-up

[cedric_renauld]

Yes between 7 to 10 Gb, to improve update possibility

[TMADOCTHOMAS]

Hey that's a great tip @Ontapforrum , thank you! I will try that when I rebuild the simulator this time.

[TMADOCTHOMAS]

Thanks @cedric_renauld !  Makes sense.