vsim 9.7 and 9.9.1 cluster perr partial

[adminqskills]

Hi everybody,

we do have the problem that we are unable to build a relationship between two, 2-Node vsim cluster.

We are only get 

sim19c51::> cluster peer show
Peer Cluster Name Cluster Serial Number Availability Authentication
------------------------- --------------------- -------------- --------------
sim19c50 1-80-000011 Partial ok

I made ipspaces, broadcast domains mit mtu 1500 as well as 9000 but still no luck.

ipspace

sim19c51::> ipspace show
(network ipspace show)
IPspace Vserver List Broadcast Domains
------------------- ----------------------------- ----------------------------

intercluster
intercluster intercluster

broadcast domain

sim19c51::> broadcast-domain show
(network port broadcast-domain show)
IPspace Broadcast Update
Name Domain Name MTU Port List Status Details
------- ----------- ------ ----------------------------- --------------

intercluster
intercluster 9000
sim19c51-02:e0g complete
sim19c51-01:e0g complete

network interfaces

intercluster
icl1 up/up 10.19.51.110/8 sim19c51-01 e0g true
icl2 up/up 10.19.51.111/8 sim19c51-02 e0g true

ports

Node: sim19c51-01
Speed(Mbps) Health
Port IPspace Broadcast Domain Link MTU Admin/Oper Status
--------- ------------ ---------------- ---- ---- ----------- --------
e0g intercluster intercluster up 9000 auto/1000 healthy

Node: sim19c51-02
Speed(Mbps) Health
Port IPspace Broadcast Domain Link MTU Admin/Oper Status
--------- ------------ ---------------- ---- ---- ----------- --------

e0g intercluster intercluster up 9000 auto/1000 healthy

we made the cluster peer command with both ICL IP and only one ICL IP

cluster peer create -address-family ipv4 -peer-addrs 10.19.50.110,10.19.50.111 -ipspace intercluster

but still show partial

sim19c51::> cluster peer show
Peer Cluster Name Cluster Serial Number Availability Authentication
------------------------- --------------------- -------------- --------------
sim19c50 1-80-000011 Partial ok

has anyone an idea what is wrong or missing? 

thanks in advance for the answers

[Ontapforrum]

Could you share any events?
::>event log show

[adminqskills]

onsim19c51::> event log show
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
8/23/2021 13:46:45 onsim19c51-01 ALERT cpeer.unavailable: Peer cluster sim19c50 is no longer available.
8/23/2021 13:46:45 onsim19c51-02 ALERT cpeer.unavailable: Peer cluster sim19c50 is no longer available.
8/23/2021 13:46:40 onsim19c51-01 ERROR cpeer.psk.cluster.revoked: ONTAP received a TLS request to authenticate with the pre-shared key of remote cluster sim19c50, but the relationship with the remote cluster has been revoked by the administrator of this cluster. The request was sent from address 10.19.50.110 in IPspace intercluster.

[Ontapforrum]

I am not sure how it suppose to show on a simulator. But, could you quickly go through the checks to ensure everything is setup ok.

Prerequisites for cluster peering:
http://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.exp-clus-peer%2FGUID-729E6FCF-9640-47B8-8C20-EC0B89F8129B.html

Creating peer:
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Protection_and_Security/SnapMirror/How_to_create_a_cluster_peer

cluster peer ping:
https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-cmpr-910%2Fcluster__peer__ping.html

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Protection_and_Security/SnapMirror/The_%22cluster_peer_health_show%22_intermittently_displays_%2...

[adminqskills]

thanks for the replies.

setup should done right, but cluster peer ping says ok , health not

cluster peer ping

onsim19c51::> cluster peer ping
Node: onsim19c51-01 Destination Cluster: onsim19c51
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
onsim19c51-01 10.19.51.110 1 64 0.05 interface_reachable
onsim19c51-02 10.19.51.111 1 64 0.183 interface_reachable
Node: onsim19c51-01 Destination Cluster: sim19c50
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
sim19c50-01 10.19.50.110 1 64 0.22 interface_reachable
sim19c50-02 10.19.50.111 1 64 0.229 interface_reachable
Node: onsim19c51-02 Destination Cluster: onsim19c51
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
onsim19c51-01 10.19.51.110 1 64 0.14 interface_reachable
onsim19c51-02 10.19.51.111 1 64 0.038 interface_reachable
Node: onsim19c51-02 Destination Cluster: sim19c50
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
sim19c50-01 10.19.50.110 1 64 0.152 interface_reachable
sim19c50-02 10.19.50.111 1 64 0.201 interface_reachable
8 entries were displayed.

cluster peer health

onsim19c51::> cluster peer health show
Node Cluster-Name Node-Name
Ping-Status RDB-Health Cluster-Health Availability
---------- --------------------------- --------- --------------- ------------
onsim19c51-01
sim19c50 sim19c50-01
Data: unreachable
ICMP: interface_reachable true true false
sim19c50-02
Data: interface_reachable
ICMP: - true true true
onsim19c51-02
sim19c50 sim19c50-01
Data: interface_reachable
ICMP: - true true true
sim19c50-02
Data: unreachable
ICMP: interface_reachable true true false
4 entries were displayed.

[Ontapforrum]

Just for an experiment , I tried to create a cluster peer between my two Simulators 9.1 & 9.7, and it worked. That means there must be some communication issue between the two clusters in your case. I have attached the screenshot just for reference purpose.  Please note, my setup is very simple, everything runs in the default ipspace.  In my example - 9.1 is a single node cluster simulator and 9.7 is a two node cluster simulator.

[adminqskills]

is it possible that duplicate serialnnumbers are the reason? I can peer two single node cluster. 

This is what i get in the log:

8/24/2021 13:32:41  onsim19c51-01    ALERT         cpeer.unavailable: Peer cluster sim19c50 is no longer available.

8/24/2021 13:32:41  onsim19c51-02    ALERT         cpeer.unavailable: Peer cluster sim19c50 is no longer available.

8/24/2021 13:32:36  onsim19c51-01    ERROR         cpeer.psk.cluster.revoked: ONTAP received a TLS request to authenticate with the pre-shared key of remote cluster sim19c50, but the relationship with the remote cluster has been revoked by the administrator of this cluster. The request was sent from address

Cluster peer ping:

onsim19c51::> cluster peer ping
Node: onsim19c51-01 Destination Cluster: onsim19c51
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
onsim19c51-01 10.19.51.110 1 64 0.054 interface_reachable
onsim19c51-02 10.19.51.111 1 64 0.251 interface_reachable
Node: onsim19c51-01 Destination Cluster: sim19c50
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
sim19c50-01 10.19.50.110 1 64 0.177 interface_reachable
sim19c50-02 10.19.50.111 1 64 0.18 interface_reachable
Node: onsim19c51-02 Destination Cluster: onsim19c51
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
onsim19c51-01 10.19.51.110 1 64 0.202 interface_reachable
onsim19c51-02 10.19.51.111 1 64 0.041 interface_reachable
Node: onsim19c51-02 Destination Cluster: sim19c50
Destination Node IP Address Count TTL RTT(ms) Status
---------------- ---------------- ----- ---- ------- -------------------------
sim19c50-01 10.19.50.110 1 64 0.178 interface_reachable
sim19c50-02 10.19.50.111 1 64 0.239 interface_reachable

[Fabian1993 · Accepted solution]

Yes, duplicate Serialnumbers will not work.

[StevenFoo]

Any idea how do we fix that, we recently upgrade one site of the box which have cluster peer to 9.9.1 and it now show same serial number for the other side (9.7) which was working before.

How do we fix this issue?

[adminqskills]

ok thanks for the information. But this behavior came with 9.7?  With 9.6 we did not have these problem.