ndmpcopy from multiple sources to same destination folder fails

[eladgotfrid1]

We have 3 NetApp storages that face the same problem when we use them as ndmpcopy destinations. In summary the issue looks like this:

When we use ndmpcopy to dump simultaneously different files from multiple sources to the same folder on a specific volume all of them but one (the one that started 1st) are failing with the following error:

Ndmpcopy: Starting copy [ 2586 ] ...

Ndmpcopy: 10.250.1.190: Notify: Connection established

Ndmpcopy: 10.13.4.65: Notify: Connection established

Ndmpcopy: 10.250.1.190: Connect: Authentication successful

Ndmpcopy: 10.13.4.65: Connect: Authentication successful

Ndmpcopy: 10.13.4.65: Log: Session identifier: 55018

Ndmpcopy: 10.13.4.65: Log: Session identifier for Restore : 55018

Ndmpcopy: 10.250.1.190: Log: DUMP: creating "/vol/NFS_Prod1Envs_EntApp_LC3/../snapshot_for_backup.29777" snapshot.

Ndmpcopy: 10.250.1.190: Log: DUMP: Using Partial Volume Dump of selected subtrees

Ndmpcopy: 10.250.1.190: Log: DUMP: Date of this level 0 dump: Mon Jul 19 09:19:47 2021.

Ndmpcopy: 10.13.4.65: Log: RESTORE: Error: Another restore is in progress on the same destination.

Ndmpcopy: 10.250.1.190: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: 10.250.1.190: Log: DUMP: Dumping /vol/NFS_Prod1Envs_EntApp_LC3/QTree/new/test-flat.file2 to NDMP connection

Ndmpcopy: 10.250.1.190: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: 10.250.1.190: Log: DUMP: Reading file names from NDMP.

Ndmpcopy: 10.250.1.190: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: 10.250.1.190: Log: DUMP: estimated 3234535 KB.

Ndmpcopy: 10.250.1.190: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: 10.250.1.190: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: 10.13.4.65: Log: RESTORE: RESTORE IS ABORTED

Ndmpcopy: 10.250.1.190: Log: DUMP: Network communication error

Ndmpcopy: 10.13.4.65: Log: DATA: Operation terminated: EVENT: INTERNAL ERROR (for NFS_Prod1Envs_SSD_LC1)

Ndmpcopy: 10.13.4.65: Connection halted: HALT: Internal error!

Ndmpcopy: 10.250.1.190: Log: DUMP: DUMP IS ABORTED

Ndmpcopy: 10.250.1.190: Log: DUMP: Deleting "/vol/NFS_Prod1Envs_EntApp_LC3/../snapshot_for_backup.29777" snapshot.

Ndmpcopy: Notify: Transfer failed

Ndmpcopy: Done

What baffles us is the fact that we have several other older NetApp storages on which this issue doesn’t occur and the ndmpcopy of multiple files, from multiple sources to the same destination folder works without any issue.

Few technical details:

1. ONTAP versions of problematic NetApps: 9.7P12, 9.7, 8.2.5
2. ONTAP versions of few none-problematic NetApps: 8.1.3P3
3. QTree is used on all the volumes
4. Ndmp version 4 is used on all
5. We use SVM-mode ndmp on the CDOT NetApps (tests with node-scope showed same results)

We will appreciate it if you share any knowledge or ideas with us on how this issue can be resolved. Will be happy to provide any technical details or configuration specs if that can help. 

Thank you!  

[eladgotfrid1]

Clarification:

9.7P12, 9.7, 8.2.5 are the NetApps that host the destination folder

Source NetApps are any others including the 9.7P12, 9.7, 8.2.5

[ttran · Accepted solution]

Hello @eladgotfrid1 ,

The functionality to perform multiple ndmpcopy restores to the same location was intentionally disabled starting with ONTAP 8.2+, which is why you haven't seen the issue with the systems running ONTAP 8.1. 

Regards,

Team NetApp

[eladgotfrid1]

Thank you.

Can this behavior be overridden somehow?

Can you point me to the doc/kb mentioning this?

[amccullo]

Hello,

The functionality was disabled in Data ONTAP 8.2 and later due to the following bug:

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=611765

I am not currently aware of any plans to reintroduce the ability to perform multiple concurrent restores to the same volume in future versions of ONTAP.

Regards, 

Team NetApp