Big problems with setting up snapmirror for buckets

[Maurice-FreeOnes]

Hello,

I have big problems with setting up snapmirror for buckets. I did spend a huge amount of time on this and it is still not working. I am pretty sure that I followed all steps and that all certificates are right now. In the jobs log overview I even see the bucket being created and after a while deleted again:

I also tried to do bucket snapmirrors within the same cluster, so no intercluster, but even there it doesn't want to work.

Running Version 9.11.1RC1 on all clusters.

I can't wait to hear the solution after spending all these hours on this issue ;(

thank you

Maurice

[Ontapforrum]

As you have already spent much time around this, I would suggest contact Support (log a ticket) as they can remote-in and better understand your environment, or point you in the right direction.

In the mean time, there are couple of pdfs that you may want to take a look just to ensure all the requisites and steps are met (In case you haven't see them).
https://www.netapp.com/media/17219-tr4814.pdf

https://www.netapp.com/pdf.html?item=/media/17229-tr4015pdf.pdf

Have you tried doing 'Packet trace' to see if there are any issues with certificates ?

[Maurice-FreeOnes]

I had an open ticket at NetApp that for our first 3 FAS2750 system the SnapMirror Synchronous license was missing. It was not cleared yet and didn't know it was this specific license which was missing. It seems like that this is the cause of all the troubles of what I found out in the logs.

[Maurice-FreeOnes]

I got the licenses right after quite some time and thought it would work after that. However it is still not working. Error message:

Operation with Snapmirror bucket relationship failed. Reason: Connection unavailable

I have disabled http (443) on both S3 SVM's so it can't be a certificate problem.

Anybody who can finally help me out?

thank you

Maurice

[Ontapforrum]

Please check this kb:

Creation of SnapMirror relationship between two S3 buckets fails with Connection unavailable:
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Protection_and_Security/SnapMirror/Creation_of_SnapMirror_relationship_between_two_S3_buckets_fa...

[Maurice-FreeOnes]

I did already. The causes in the document:

• S3 SnapMirror will automatically use HTTPS for management when enabled on the source and/or destination S3 object store server
• Misconfigurations in the security certificates of the source and/or destination data SVM or installing the wrong certificate cause "connection unavailable" errors
• InterCluster LIFs on source and destination must also be able to connect to the source and destination S3 server data LIFs for the connection to succeed

Reason 1 can't be it, because I also tried it without https.

Reason 3 can't be it, because all ping tests we did arrive.

Reason 2, I first thought can't be the reason because like I said I disabled https. However when I read for the 30st time thru this document I thought that I maybe still would need this certificate installed when only using http. Can you confirm this?

When I tried the solution: vserver object-store-server create -vserver s3secondary -object-store-server s3secondary.fqdn.com -certificate-name s3secondary.fqdn.com

I got a dash (-) as result at certificate-name field for the specific S3 SVM. Howeve when I tried to install the certificate I got this as result:

"Enter certificates of certification authorities (CA) which form the certificate chain of the server certificate. This starts with the issuing CA certificate of the server certificate and can range up to the root CA certificate.

Do you want to continue entering root and/or intermediate certificates {y|n}: n

Error: command failed: duplicate entry"

Hope somebody can help me finally solve this problem.

thank you!

[Maurice-FreeOnes · Accepted solution]

I got it solved, one of the problems was the firewall pff

[MateenMohammed1]

Hello Maurice,

I am also facing similar issue, I checked everything all configurations looks ok. Even if I try to setup S3 Snapmirror for the Bucket within same SVM I get the same error message. 

I see you got it solved, what was the problem related to firewall?

[Alexander2]

Hello all,

I am also having the exact same issue. Does anyone have any information on how to solve the problem?

[MateenMohammed1]

I solved it, it was firewall issue. You need to open port 443 among all the three lifs ( S3 Lif, local Intercluster Lifs and Remote Intercluster Lif). Like Mesh Connection among the 3 lifs.

https://kb.netapp.com/onprem/ontap/dp/SnapMirror/ONTAP_S3_to_ONTAP_S3_SnapMirror_failing_with_connection_unavailable

[Alexander2]

Hello,

thanks for your reply. However I still get the same error message.

I tried to add a S3 Snapmirror for a bucket with an SVM and only enabled http to rule out any certification problems.

Both the intercluster lif and the data lif for the S3 server have the

management-http

and

management-https

service added.

I also checked all the other suspects mentioned above. Unfortunately I still get the Connection unavailable (2) error.

A Snapmirror between volumes does work on the SVM.

[MateenMohammed1]

Please check this KB

https://kb.netapp.com/onprem/ontap/dp/SnapMirror/S3_SnapMirror_creation_failing_with_connection_unavailable

In my case S3 Data Lif, Source Intercluster Lif and Destination Intercluster Lif were in different subnets.

For Example:

Source SVM

S3 Data Lif:  10.0.32.21

Source Intercluster Lif1: 172.16.0.21.

Source Intercluster Lif2: 172.16.0.22.

Destination SVM

DestinationS3 Data Lif:  10.0.25.10

Destination Intercluster Lif1: 172.18.0.22.

Destination Intercluster Lif1: 172.18.0.23.

I had to get Firewall Team involved to open ports 

Source Data Lif ---- open 443 to Destination Data Lif, Local Intercluster Lif1,Local Intercluster Lif2, Destination Intercluster Lif1,Destination Intercluster Lif2

Destination Data Lif ---- open 443 to Source  Data Lif, Local Intercluster Lif1,Local Intercluster Lif2, Destination Intercluster Lif1,Destination Intercluster Lif2

Local Intercluster Lif1 ---- open 443 to Source  Data Lif, Destination Data Lif, Local Intercluster Lif1, Local Intercluster Lif2, Destination Intercluster Lif1,Destination Intercluster Lif2

So on....

All the LIF's should be allowed to communicate via 443.

[Alexander2]

Hello,

So, that means for an (intracluster) S3 SnapMirror relationship the Data LIF and Intercluster LIF have to be able to communicate?

For volume-level SnapMirror relationships that is not the case: https://kb.netapp.com/onprem/ontap/dp/SnapMirror/What_LIFs_are_used_for_intracluster_SnapMirror_volume_level_relationship