(RPC: Timed out) when creating volumes

[VJAYRAGHAVAN]

Hi,

We are getting RPC timeout errors when we try to create new volumes.  Please find the error message when i try to create the volume 

N0905800::*> volume create -vserver N905800-T-PII -volume vj_test_rpc_time_out_error -aggregate N0905800_02_D0905010_01 -size 20GB -state online -type RW -space-guarantee none -percent-snapshot-space 0 -unix-permissions 775 -snapshot-policy none -user MQM -group MQM
[Job 25861] Operation cleanup

Error: command failed: [Job 25861] Job failed: RPC: Timed out

We I check the EMS log it looks like all the LS mirrors are failing with the below error message.

error="Volume config database write failed (RPC: Timed out)."

I have checked my cluster quorum and I am not seeing any issues. 

I have also tried to create volumes in different Nodes and different Vserver and i still get the error.

Has anyone seen this error before? Can someone help me to fix it.

We are running NetApp Release 8.2.1P3 Cluster-Mode

Regards,

Vijay

[JSHACHER11]

sure looks like an RDB replication issue to me

[georgevj]

You may check the output of "cluster ring show" from advanced privilege mode to see if an RDB ring is healthy or not.