nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Root volume is damaged

bashirgas

Hi,

I have two node cluster (vsim 8.3) running on VMware Fusion. Doing some exercise I changed the MTU on my cluster ports using network port broadcas.... after few minutes one of the nodes went down and when it came back again, the following message appeared.

***********************
** SYSTEM MESSAGES **
***********************

The contents of the root volume may have changed and the local management
configuration may be inconsistent and/or the local management databases may be
out of sync with the replicated databases. This node is not fully operational.
Contact support personnel for the root volume recovery procedures

After a while the other node also showed same message.

I can not see my root volume in clustershell and I cannot use any of the subcommands of vol/volume....

Anyone who had a similar problem with VSIM 8.3?

How can I recover my root volumes or repair it in clustershell?

If I enter a nodeshell, I have no option creating additional volumes using the vol command...

I'm stuck..

Thank you

Bash

Find more posts tagged with

ONTAP Simulator

Accepted answers

SeanHatfield

OK. Need to try to revive the RDBs. Shut down the non-epsilon node, halt the epsilon node and boot to the loader. unset the boot_recovery bootarg and see if it will come back up.

unsetenv bootarg.init.boot_recovery

If the epsilon node comes back up, make sure the cluster ports are mtu1500, then try to bring up the other node.

All comments

Emile-Bodin

you can connect to the node management ip and run

::> node run local

This will drop you down the node shell and you should be able to manager the volumes 7-mode style.

cmemile01::*> node run local
Type 'exit' or 'Ctrl-D' to return to the CLI
cmemile01-01> vol status
Volume State Status Options
vol0_n1 online raid4, flex root, nosnap=on, nvfail=on 64-bit

What do you get running "::> cluster show" from the cluster shell?

If you get RPC errors and the cluster can not communicate over the interconnect the RDB is out of sync, you try and undo the changes that you have made on the MTU.

::> net port modify -node cmemile01-01 -port e0a -mtu 1500

Another option is to redeploy the sims and start fresh.

bashirgas

I can access the nodeshell and the output looks ok

n2> vol status vol0
Volume State Status Options
vol0 online raid_dp, flex root, nvfail=on
64-bit
Volume UUID: b450b0c7-fe1f-4d49-adf5-160776524770
Containing aggregate: 'aggr0_n2'

The vol0 has no space issues

n2> df -g vol0
Filesystem total used avail capacity Mounted on
/vol/vol0/ 7GB 1GB 5GB 19% /vol/vol0/

n2> vol
The following commands are available; for more information
type "vol help <command>"
autosize lang options size
container offline restrict status
destroy online

I cannot create new volumes inside nodeshell

clu1::> aggr show -fields has-mroot
aggregate has-mroot
--------- ---------

Warning: Only local entries can be displayed at this time.
aggr0_n2 true
n2aggr false
2 entries were displayed.

The Volume subcommands are not available

clu1::*> volume ?

Error: "" is not a recognized command

clu1::*> volume

Only the node lif is available

clu1::*> net int show
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
clu1
c1n2mgmt up/up 192.168.101.37/28 n2 e0c true

cluster show is not available

clu1::*> cluster setup

Error: command failed: Exiting the cluster setup wizard. The root volume is damaged. Contact support personnel for the
root volume recovery procedures. Run the "cluster setup" command after the recovery procedures are complete.

I changed the MTU on both nodes in the cluster but something caused cluster configuration lost, I cannot figure it out!

clu1::*> net port show
(network port show)
Speed (Mbps)
Node Port IPspace Broadcast Domain Link MTU Admin/Oper
------ --------- ------------ ---------------- ----- ------- ------------
n2
e0a Cluster - up 1500 auto/1000
e0b Cluster - up 1500 auto/1000
e0c Default - up 1500 auto/1000
e0d - - up 1500 auto/1000
e0e - - up 9000 auto/1000
5 entries were displayed.

In advanced or Diagnostic level, I cannot run cluster ring show. Its like cluster services are not available!.

Never seen this before.

SeanHatfield

I think you've hit on an interesting scenario. Jumbo frames don't work on VMware Fusion, and by attempting to use jumbo frames for the cluster network something has gone wrong with the RDBs.

Did they panic and reboot?

Do you know which one had epsilon?

bashirgas

That's what I have noticed. All 5 RDBs are gone but somehow the cluster shell remains!!!

No panic, but the node2 (the one with the epsilon role) rebooted.

haopengl

1. Bring the node to the LODER prompt:

::*>halt -node <node>

2. Check to see if the following bootargs have been set:

LOADER>printenv bootarg.init.boot_recovery<br>LOADER>printenv bootarg.rdb_corrupt

3. IF either bootarg as been set to a value, un-set it and boot ONTAP

LOADER>unsetenv bootarg.init.boot_recovery<br>LOADER>unsetenv bootarg.rdb_corrupt<br>LOADER>bye

SeanHatfield

The simloader has no saveenv, so when you bye none of the changes will be saved. Instead you have to boot, and let it proceed to at least the boot menu for changes from the loader prompt to be persistent.