Copying millions of small files from Windows server to NetApp ontap

[admin2go]

• Hello Everyone,

we are trying to copy tens of millions of small files to Ontap but the performance of Robocopy and any other tools that used SMB will be slow due to SMB being the issue. 

is there a way to use  SDK to push the files to ONTAP by APIs? Would that be quicker? Can we preseve NTFS permissions?

please let me know your thoughts. 

[Sergey_Osipov · Accepted solution]

https://xcp.netapp.com/

[Drew_C]

^ this is the answer.

[SpindleNinja]

I"ll second XCP as the correct way to do this, but I'm curious, can you expand on what your thoughts on using the API? 

[admin2go]

I am familiar with xcp but the issue we have is not the data size. The issue is we have 10s of millions and sometimes hundreds of millions of file. 

Since XCP and other tools use SMB, SMB has a major obstacle with small files. 

For instance, a single 1GB file can take 3 seconds. Where if you have 1 million files that total to 1GB could take up to 8 hours.

I was hoping that NetApp has some interface where we can:

1) push the data to copy through API interface that is robust so we bypass SMB.

2) copy ACL (NTFS Permissions)

This way we can get a much faster throughput and those 8 hours for one million files would take 45 minutes or so due to SMB overhead is removed.

We can use SDK for ONTAP if such an option is possible.  Just need to make sure the API is robust, and offers ACL copy.

Thoughts?

[SpindleNinja]

Not at that level.  that almost sounds like an FLI type thing but for NAS 🙂 

I've had customer use XCP to move millions of files no problems, 

https://www.netapp.com/media/20029-tr-4863.pdf.