Data ONTAP Discussions

Converting regular folders to Qtrees

I recently ran into a scenario where large amount of data (lots of files across lots of sub-folders) had to be moved from a regular folder into a Qtree (in the same volume) for quota purposes.

 

Challenge was how to get the data moved between the two folders as quickly as possible with as little impact to snapshot space consumption. We tried many approches (File Explorer, Robocopy, fancy multi-threaded / multi-job code), and eventually found that the fastest way to move the data was with a single line PowerShell command:

 

Get-ChildItem -Path <path_to_folder_1> -Recurse | Move-Item -Destination <path_to_folder_2>

 

That's it. Moving hunderds of thousands of files acorss very complicated nested sub-folder structure took only a few seconds with about 2% impact to snapshot space.

 

Thought I would share in case you find yourself in similar situation.

5 REPLIES

Re: Converting regular folders to Qtrees

Yaron,

 

It's your friends from NJ....

 

Believe it or not, it's faster to move with robocopy wrapped in powershell... 

Re: Converting regular folders to Qtrees

May be so, but robocopy doesn't really move files at the metadata level. It copies and deletes, meaning the snapshot space will explode.

 

Hope you guys are surviving the cold weather...

Re: Converting regular folders to Qtrees

Yeah it's cold up here for sure... But I would love to see your test case, b/c I can't see how your single line is faster...   

Re: Converting regular folders to Qtrees

Remeber, the use case is moving files between two folders (one regular and one qtree) in the same volume. The PowerShell command will not copy any data but only update the metadata of the files. Robocopy will copy the entire data and delete the source.

Re: Converting regular folders to Qtrees

Now I see your point... I'll email you for more details..

Forums