[Cloud Sync] troubleshooting

[kiban-unyo]

Hi,

I transferred from on-premises NFS to AWS EFS.

  NFS---DataBroker---AWS-DX---AWS EFS
Please tell me about the following.

[1]

The permission of the folder has changed in the source and the destination.
The 'write permission' disappeared from the destination folder.
Is synchronization of folder/file permission guaranteed?

[2]
What is the cause of the "Scan Failed" or "Copy Failed"?
Since retransmission is successful, permission is not a problem.
<sample log>
{"path":"request-2018-11-10.log","type":"FILE","size":21970602,"mode":33188,"uid":0,"gid":0,"mtimeSec":1541862011} with error {"errno":-5,"code":"EIO","syscall":"open",　"path":"mnt/5bfd09f393141b000870d5fc/21dd94f1-69d1-414d-b363-66bfdc64abf0_src/qtr_fnf001/log/request-2018-11-10.log"}

[3]
Traffic(from DataBroker to Internet) was generated during transfer.
What kind of processing is being carried out concretely?(Directory structure of the transfer source?)
and can I narrow down the traffic?

Regards

[yaronh]

Dear sir,

As agreed with the team we will work on these topics via emails. I will, however, share a concise reply here as well.

[1] Permissions of folders in NFS can, on occasion, be different from source. This has been identified as a bug and we are working on solving it. Additional sync operation may fix that issue, but not with 100% certainty.

I will have an update with timeline for fix soon, but we gave it a high priority. 

[2] Scanning happens before the copying. When the error is scan failure it is most likely failing to read from the source.
It may happen intermittently, but as you mentioned, the next sync will retrasmit that.

We did see that happen when targetting EFS as EFS can be slow to react (Depending on what is the EFS configuration.

[3] The broker has internet access and is sending meta data to the backend of the SaaS service of CloudSync.
It sends messages to the SQS queues that are critical for the sync operation. 

Your data is NOT being sent to the backend but is copied directly.

Other kind of traffic are Kibana artifacts that allow us to determine the status of the transfer.
Potentially this can be reduced, but we don't recommend it.

I will send the information and more over email as well.

Best,

Yaron Haimsohn

Manager, Cloud Solution Architecture

NetApp

yaron.haimsohn@netapp.com

[kiban-unyo]

Hi,

[1]
- Under what conditions will the bug occur?
- Is there a workaround for the bug?
- Cloud you tell me when the bug will be fixed? I decide whether to use cloudsync depending on the schedule.

[2]
>When the error is scan failure it is most likely failing to read from the source.
Cloud you tell me specific conditions of scan failure?

>We did see that happen at other customers when targeting EFS as EFS can be slow to react
 Is the above condition an example of copy failure?
 Cloud you tell me what other conditions of copy failure are there?

[3]

>Other kind of traffic are Kibana artifacts that allow us to determine the status of the transfer.
>Potentially this can be reduced, but we don't recommend it.
 If I do the above, what settings should I change? Also what is the impact at that time?

Regards

[yaronh]

Hello Keisuke,

1] The bug appears on some directory creations, where the directory ends up getting the permission of the first
file copied in that folder.

There is no workaround, but subsequent sync can see a mismatch in the folder permission and can fix that.
That subsequent action will fix the permission on high likelihood - It will not recopy files just fix permissions.

We are working on a fix that should be available for next mid-month drop.

2] This is an intermittent failure. Your source returned an error when we tried to read the file.

It may be a busy source, slow source, network timeout etc. I can't be mucn more specific as i need to relate
to a specific error. The one you provided was a read timeout.

3] The impact would be mostly on the backend service event logging, and our ability to look at some of your
issuse and offer assistance with it.

So, we do not recommend it, but as an end user the service will have no ill effect.

The step would be to not allow the broker to get to the kinesis service in AWS.

Note: SQS is mandatory and blocking it will block your ability to sync.

Best,

Yaron