NetApp Community Update
This site will enter Read Only mode on July 23 as we prepare to move to a new platform. You will still be able to view content, but posting and replying will be temporarily disabled.
We're excited to launch our new Community experience on July 30 and more information will follow soon.
Stay connected during the transition - Join our Discord community today.

Network and Storage Protocols

Dirs with large number files (100k) too slow for backup - NDMP OK??

JM01JM01
4,011 Views

Problem :  Backing up dirs with 100,000 small files fails.  Using NetBackup and backing them up as NFS mounts on the Linux box  times out. Tested NetBackup timeout settings extensively they still fail. Listing an individual file is OK but listing the entire contents suing the ls command is very slow  [ 7-10 mins].  This explains why the backups fail. Backing up the same data using NDMP via NetBackup is a respectable ~100MB/s. 

My understanding NetApp and a 100,00 small files while always exhibit this slow behaviour - true? Any technotes on this?      

1 REPLY 1

billshaffer
4,011 Views

I wouldn't expect this for only 100K files, but the behavior is pretty typical, and not at all unique to NetApp.  If you put a bunch of small files anywhere you'll get similar results.  It's an issue with the filesystem trying to scan the number of inodes.

Bill

Public