Estimating the File/Folder count on a CIFS volume

[darraghos]

Guys, I have a large CIFS volume (2TB+) with snapshots enabled and lots of files/directories. I'd like to try and estimate how many of each I have. Is there any accounting for this on the cluster? Perhaps the iNode count or the like? I'd rather not try and run something like TreeSize against it! Will take months!

[TMAC_CTG]

Have a look at NetApp's XCP : 

https://xcp.netapp.com

There are some options to XCP to allow you to do what you ask. Plus it is pretty fast

[darraghos]

Thanks! Looks cool. Since it run client side though will it not be just trawling the CIFS volume over the network or does it do something clever with an API or the like?

[TMAC_CTG]

Please have a look through the README and Docs:

https://mysupport.netapp.com/documentation/docweb/index.html?productID=63525&language=en-US

Specifically: 

Core Engine Innovations (XCP SMB Features)

 Supports Windows, CLI only
 Extreme performance (~25x comparable tools)
 Multiple layers of granularity (qtrees, subdirectories, criteriabased filtering)
 Easy deployment (64-bit Windows host-based software)

[darraghos]

Thanks. Still not sure if the x25 perf increase is due to multi-threading client side or some specific integration though. Since my last scan took ~ 40 hrs a 25 times perf increase would be awesome!

[TMAC_CTG]

It may be that since NetApp wrote XCP and they wrote the CIFS server you are trying to query, there may be some built-in efficiencies taking advantage of the protocol

[mbeattie]

Hi,

If you just want to get a file count on the volume including snapshots...

cluster1::> volume show -vserver vserver1 -volume cifs_data_001 -fields files-used
vserver  volume        files-used
-------- ------------- ----------
vserver1 cifs_data_001 118

/Matt

[darraghos]

Thanks! Unfortunately I need to exclude snapshots as want the 'live count but thanks for the info. 

[parisi · Accepted solution]

The perf increase is due to multithreading.

I'd suggest using the NFS version to scan though. It's a bit more robust and, honestly, works a little better than the SMB version. Set up a Linux VM and install XCP there. In XCP 1.6, we have File Systems Analytics, which does a regular poll of the file system to report on things like file counts. This is readonly and won't change any data.

https://blog.netapp.com/xcp-data-migration-software

https://whyistheinternetbroken.wordpress.com/2018/07/02/how-to-find-average-file-size-and-largest-file-size-using-xcp/ 

https://whyistheinternetbroken.wordpress.com/2020/07/28/xcp-161/

https://whyistheinternetbroken.wordpress.com/2019/08/30/using-xcp-to-delete-files-en-masse/

[darraghos]

Thanks for the information guys. Multi-threading is still a big boost I'd imagine so I will test and see. It's a pity we can't get at Master File Table and use something like TreeSize. 

[parisi]

I believe TreeSize would work, but it would likely crawl the entire filesystem over the protocol and that would be single threaded.