nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Reason : Failed to perform create op for bucket --> Reason: Unknown. [139591706]

Maurice-FreeOnes

It took me very long to setup S3 SnapMirror Bucket in our clusters. However in the end I was able to make it work. I created some test buckets and it worked and i was very happy. However when I tried to sync our S3 buckets from 1 cluster to the other all kind of stuff started to popup with strange and Unknown error reasons. Here is one example:

ch-strt-cl-001::*> snapmirror create -source-path ch_fx_svm_s3_001:/bucket/video-generated -destination-path of_fx_svm_s3_001_dest:/bucket/video-generated-dest4

[Job 78175] Job is queued: Create an S3 SnapMirror relationship between bucket "ch_fx_svm_s3_001:video-generated" and bucket "of_fx_svm_s3_001_dest:video-generated-dest4"..

ch-strt-cl-001::*> job watch-progress -id

<integer> Job ID

ch-strt-cl-001::*> job watch-progress -id 78175

Complete: Cannot create S3 SnapMirror relationship between bucket "video-generated" and bucket "video-generated-dest4". Reason : Cannot create S3 SnapMirror relationship between bucket "video-generated" and bucket "video-generated-dest4". Reason : Failed to perform create op for bucket "video-generated" in Vserver "ch_fx_svm_s3_001". Reason: Unknown. [139591706]

Hope that somebody can help me out.

thank you.

Maurice

Find more posts tagged with

Accepted answers

Maurice-FreeOnes

Update: I rebooted all servers by installing the next update. After that it worked.

Now I am at the next problem. I need to sync a bucket of around 60 TB from USA to Europe, but the sync is going very unstable. Sometimes it's running and with around 1.5 Gib/s and then the other time nothing happens for like an hour.

FabricLink Status: 67d overdue: 2009271 objects to transfer, pushing 45723.8GB; throughput 816.9MB/sec, 31 ops/sec

FabricLink Status: 67d overdue: 2009259 objects to transfer, pushing 45723.1GB

Pretty disappointing 😞
I also have some buckets which are already fully synced and they do not seem to have this problem.

All comments

RossC

Hi @Maurice-FreeOnes

If the errors you are seeing here don't get any results on our Knowldgebase, then I would encourage you to open up a case with our support team so they can dive into the ONTAP snapmirror logs and see what's going on there.

Maurice-FreeOnes

Unable to delete a snapmirror bucket relationship in ONTAP.

of-strt-cl-001::*> snapmirror del -destination-path of_fx_svm_s3_002:/bucket/tube-network -source-path of_fx_svm_s3_001:/bucket/tube-network

Warning: The operation will delete the S3 SnapMirror relationship between bucket "of_fx_svm_s3_002:tube-network" and bucket "of_fx_svm_s3_001:tube-network".
Do you want to continue? {y|n}: y
[Job 75456] Job failed: Failed to delete Snapmirror relationship on bucket "tube-network" and vserver "of_fx_svm_s3_001". Reason: Failed to perform delete op for bucket "tube-network" in Vserver "of_fx_svm_s3_001". Reason: Unknown. Run the (privilege: advanced) "snapmirror delete" command to cleanup the stale SnapMirror relationship. Run the command "snapmirror delete" again using vserver "of_fx_svm_s3_001" and bucket "tube-network" as destination.

Error: command failed: [Job 75456] Job failed: Failed to delete Snapmirror relationship on bucket "tube-network" and vserver "of_fx_svm_s3_001". Reason: Failed to perform delete op for bucket "tube-network" in Vserver
"of_fx_svm_s3_001". Reason: Unknown. Run the (privilege: advanced) "snapmirror delete" command to cleanup the stale SnapMirror relationship. Run the command "snapmirror delete" again using vserver "of_fx_svm_s3_001" and bucket
"tube-network" as destination.

of-strt-cl-001::*>

SnapMirror commands like Abort, break etc are all not valid for Buckets. And when delete is not working you can't do anything. The whole SnapMirror Bucket thing in NetApp seems to be a joke to me. Very sad because like I said I put a lot of work in it to make it (kind of) work. All my 18 NetApp nodes did run out of support at the end of last year and I do not have a budget at the moment to renew.

RossC

Hi @Maurice-FreeOnes

Sorry to hear you're having a bit of trouble with this S3 Snapmirror capability.

Can you please confirm what model system you are using and what ONTAP version you are running on this system?

I can then double check our bug system to see if anything matches similar to what you are seeing.

I suspect you may be hitting some kind of bug. Looking at our Knowledgebase we do have several articles that speak about issues with deleting S3 buckets and snapmirror. One take away though as most of the solutions require support intervention due to support level diagnostics commands needing to be run.

You may wish to have a chat with your local NetApp account rep or NetApp partner, as we do have some lower-cost entitlement offers like SupportEdge Basic - https://www.netapp.com/pdf.html?item=/media/31577-NetApp-SupportEdge-Basic-Service-Description.pdf

Maurice-FreeOnes

Hi Ross, thank you for taking time for my issues.

All systems run on version 9.12.1 and are FAS2750 systems.

Our first S3 SVM's were created at the time that S3 was just launched. Maybe the problems have to do with this. But that is just a guess.

Is there any command to stop a SnapMirror Bucket sync other than the delete command? Since delete is not working for me.

I would appreciate a lot if somebody of NetApp can look into this whole thing. Like I said I already spend a tremendous time into this and I did hire 2 different NetApp consultants as well without anybody whom could solve the problems. We are already using NetApp over 10 years and it's insane stable, so I do not understand the current problems we have.

regards, Maurice

RossC

Hi @Maurice-FreeOnes

Thanks for sharing the additional info.

I poked around our bugs information and don't see anything specific that popups. This could either mean you've hit a new issue or something else we've not considered is going on.

Apologies I couldn't be more help on that. You might want to consider asking your questions on our NetApp Discord community - http://netappdiscord.com/ as sometimes community members are on our Discord but not watching this forum.

I know I mentiond it, but given you are trying to run a production workload / configuration I would encourage you to reach out to your local NetApp accounts team or partner to discuss entitlement options to meet your budget. We do have lower levels of entitlement (like Basic and some other unlisted ones you can ask about).

Maurice-FreeOnes

Hi Ross,

I have it working between to other clusters now. And there it works without any problems.

The 2 clusters where it doesn't want to work between we made some network changes and I was able again to snapmirror bucket within the same cluster and SVM without any problem. When I did this on the other cluster I got this error:

"Empty check on bucket test-dest timed out."
I also get this error when I try it to or from this cluster to the other cluster:

Complete: Unable to configure remote Vserver "ch_fx_svm_s3" for S3 SnapMirror. Reason: Empty check on bucket test timed out. [139591739]

Do you have any idea what can cause this?

thanks for your time

Maurice

Maurice-FreeOnes

Anybody else?

Maurice-FreeOnes

Update: I rebooted all servers by installing the next update. After that it worked.

FabricLink Status: 67d overdue: 2009271 objects to transfer, pushing 45723.8GB; throughput 816.9MB/sec, 31 ops/sec

FabricLink Status: 67d overdue: 2009259 objects to transfer, pushing 45723.1GB

Pretty disappointing 😞
I also have some buckets which are already fully synced and they do not seem to have this problem.

Drew_C

When in doubt, reboot! Thanks for coming back with the update.

For your follow-on issue, would you please ask in a new thread so we can wrap this one up?

You might also want to check in on our Discord to see if anyone there can assist: http://netappdiscord.com