Millions of files in a single directory

BriggsCorp — Thu, 05 Jun 2025 07:18:30 GMT

Does anybody have to deal with an application that write tons of files to a single directory? Currently I have a shared folder on my Windows file server that has 1.8 million files in it. They are all small and take up less than 100GB, but backing up and working with the folder is nearly impossible. Would moving this to a CIFS share on my NetApp 3040 give me any advantages? How would I back this up then? We don't have a second appliance to copy to and I'm assuming you can't use NetBackup against the NetApp. I'm just wondering if the NetApp would be more efficient trying to deal with all these files. Thanks for any ideas.

Re: Millions of files in a single directory

adamfox — Tue, 16 Feb 2010 15:48:23 GMT

You may get some advantage to putting it on a NetApp, but these kind of environments are a challenge for any application that needs to traverse the directory and pull metadata information (like all backups do).

Here are some thoughts on where/how NetApp may be able to help.

o NetApp snapshots are not only space efficient, but performance neutral, so for most kinds of basic restores, you can keep a good number of point in time copies for recovery purposes without having to rely on dump/restore

o You can use NetBackup with NetApp. NetApp has worked with Veritas/Symmantec for years. NDMP is the most popular way to do it. But with directories with lots of files dumps to tape will still take a long time.

o One solution customers have used is to use volume snapmirror which is a physical replication technology for NetApp to replicate that volume to either another set of disks (i.e. different aggregate) on the same controller or to a volume on a different controller. Because it is a physical copy vs a logical copy the # of files doesn't matter. If you still require tape, then you can run that backup on the 2nd copy knowing that you have a DR copy on a 2nd set of disks that can be backed up quickly and thus the backup to tape time is less critical.

o Alternatively you can rely on the local snapshots as your basic recovery, then allow the backup a longer time to run.

o You can use SnapMirror to Tape as your backup solution. This does a full backup and is a physical copy so the # of files don't matter. The downside is there is no incremental support for this.

The best solution is to break up those directories into multiple directories, but some applications don't allow that.

Hope this helps.

topic Re: Millions of files in a single directory in Network and Storage Protocols

Millions of files in a single directory

Re: Millions of files in a single directory