Data ONTAP Discussions

Converting regular folders to Qtrees

I recently ran into a scenario where large amount of data (lots of files across lots of sub-folders) had to be moved from a regular folder into a Qtree (in the same volume) for quota purposes.

 

Challenge was how to get the data moved between the two folders as quickly as possible with as little impact to snapshot space consumption. We tried many approches (File Explorer, Robocopy, fancy multi-threaded / multi-job code), and eventually found that the fastest way to move the data was with a single line PowerShell command:

 

Get-ChildItem -Path <path_to_folder_1> -Recurse | Move-Item -Destination <path_to_folder_2>

 

That's it. Moving hunderds of thousands of files acorss very complicated nested sub-folder structure took only a few seconds with about 2% impact to snapshot space.

 

Thought I would share in case you find yourself in similar situation.

6 REPLIES

Re: Converting regular folders to Qtrees

Yaron,

 

It's your friends from NJ....

 

Believe it or not, it's faster to move with robocopy wrapped in powershell... 

Re: Converting regular folders to Qtrees

May be so, but robocopy doesn't really move files at the metadata level. It copies and deletes, meaning the snapshot space will explode.

 

Hope you guys are surviving the cold weather...

Re: Converting regular folders to Qtrees

Yeah it's cold up here for sure... But I would love to see your test case, b/c I can't see how your single line is faster...   

Re: Converting regular folders to Qtrees

Remeber, the use case is moving files between two folders (one regular and one qtree) in the same volume. The PowerShell command will not copy any data but only update the metadata of the files. Robocopy will copy the entire data and delete the source.

Re: Converting regular folders to Qtrees

Now I see your point... I'll email you for more details..

Re: Converting regular folders to Qtrees

Hi yaron,

 

I tried the command but it is always failing with access denied to move the source folder.

When I create a normal folder on the volume level, I can move date with your command without any issues.

 

The NTFS permissions on the folder and the qtree are the same.

 

Do you have any idea what the issue could be?

 

Thanks and regards

Matthias

Forums