Needed a Benchmark Tool. So I Wrote one.

by New Contributor on ‎2011-04-21 01:49 PM

Had a customer that needed a very complex file system for testing a backup application and being able to contrast a file based migration versus a worst case scenerio of a file based tool. I looked at a number of tools and none of them did specifically what I needed to do. The tool should build very deep and wide directories with millions of files. So I wrote the following that creates a huge Directory tree that is X wide and Y deep with Z Files.

It estimates how many files it will create as well as how many directories as well as a rought estimate of time required to complete.

See sample output below;

PS X:\PowerShell\create1billion> .\BuildManyFilesv0.76.ps1

Supply values for the following parameters:
Files: 25
Depth: 4
Width: 3
Mass File Creator starting with following attributes
Files =  25,Depth = 4,Width = 3
Number of Files will Equal  1600
Space needed will equal  6553600  Bytes
At 50 Files/Second ; Time to Completion = 0.444444444444444  Hours
At 50 Files/Second ; Time to Completion = 0.0185185185185185 Days
At 50 Files/Second ; Time to Completion = 5.07356671740233E-05  Years
Start Time =  4/21/2011 1:59:21 PM

See Sample Code Below;

# Test program to create Many on a Drive Letter
# To run this program simply go to the drive letter you want To test against.
# Then type .\createFiles.ps1
#
# Program will create a all the container directoys that is named 1...x.
# Under that directory it will create 1...x subdirectorys deep,
# and x Directorys wide, and each directory will contain the prescribed file count.
# Each file will contain a file  named by the time down to the millisecond
# Each file will contain psuedo-random data

param( [parameter(Mandatory=$True)][validaterange(2,1024)][int]
     $Files = $(read-host -prompt "Number Of Files per Dir. (2-1024)"),
         [parameter(Mandatory=$True)][validaterange(2,100)][int]
     $Depth = $(read-host -prompt "Number of Directorys Deep (2-256)"),
         [parameter(Mandatory=$True)][validaterange(2,100)][int]
     $Width = $(read-host -prompt "Number of Directorys Wide (2-256)"),
         [parameter(Mandatory=$True)][validaterange(4096,65536)][int]
     $FileSize = $(read-host -prompt "Size of File in Bytes (4096-65536)")
     )

function simppow($a, $b)
     {   if ($b -eq 0)   {  return 1       }  
         else    { return ((simppow $a ($b-1)) * $a) }
     }

function createdir($dep, $width, $depth, [int]$NumOfFiles, $Filesize)
   {   if ($dep -eq '0')
       {    $textout = (1..$FILESIZE|% {[char[]](Get-Random -Maximum 300)}) -join ""
            1..$NumOfFiles|%{ $filename="{0:MM} Month {0:dd} Day {0:HH} Hour {0:mm} Minute {0:ss} Second {0:FFF} MS.log" -f [datetime]::Now
            $TEXTOUT | out-file -filepath $Filename }
            write-host "Created"$NumOfFiles" Files dir="$(pwd).Path" : Stamp = "(get-date)
            $supress = cd ..
            createdir ($dep+1) $width $depth $NumOfFiles $filesize
       }
      else
      {    $name=1
           while(test-path $name) { $name++ }
           if  ($name -gt $width)
           {    if ($dep -eq $depth){ return }
               $supress = cd ..
               createdir ($dep+1) $width $depth $NumOfFiles $filesize
           }
          else
          {   $supress = mkdir $name
              $supress = cd $name
              createdir ($dep-1) $width $depth $NumOfFiles $filesize
          }
      }  
   }

# Main Body

write-host "Mass File Creator starting with following attributes "
write-host "Files = $Files,Depth = $Depth,Width = $Width,FileSize=$FileSize Bytes"
$NumberOfFiles=(simppow $Depth $Width)*$Files
Write-host "Number of Files will Equal $NumberOfFiles"
Write-host "Space needed will equal "($NumberOfFiles*$Filesize)" Bytes"
if (($numberoffiles/3600/24/365) -gt 1)
   { Write-host "At 50 Files/Second ; Time to Completion ="($numberoffiles/3600/24/365)" Years"
   }

elseif (($numberoffiles/3600/24) -gt 1)
   { Write-host "At 50 Files/Second ; Time to Completion ="($numberoffiles/3600/24)"Days"
   }
else
   { Write-host "At 50 Files/Second ; Time to Completion ="($numberoffiles/3600)" Hours"
   }

write-host "Start Time = "(get-date)
createdir $depth $width $depth $Files $FileSize
write-host "End Time = "(get-date)

Comments

HPCreateData does the same with some more features. really useful - give it a try

New Contributor

Actually that was the tool I started with. I am a big fan of HPCreateData, and that is what I started with but there are the problems with HPCreateData. It can only go 10 Directories Deep, and only 10 Directories wide and we had a request to go 256 directories deep. It also can only create 1024 files in a directory, we needed 4096 files in some directorys. It also only worked against Physical Drives, and I needed to run this agains CIFS shares. With that said the HP tool does a lot that my quick and dirty tool doesnt like different patterns, different mixed file sizes and a cute graphical interface. I will revisit this source code in another few weeks when my current project is over and see if I can add those features to my script, and maybe a gui....but you know, one of the things I like about this little script is that it is so small, modular, and simple.

New Contributor

Just cleaned up the code, Added the ability to change the block size, also added proper typing on the input variables. Also fixed the random charactor generator as before it was a hack, and now its clean. Shortened the code up considerably with powershell tricks taught to me by Glenn S. Thanks Glenn.

Warning!

This NetApp Community is public and open website that is indexed by search engines such as Google. Participation in the NetApp Community is voluntary. All content posted on the NetApp Community is publicly viewable and available. This includes the rich text editor which is not encrypted for https.

In accordance to our Code of Conduct and Community Terms of Use DO NOT post or attach the following:

  • Software files (compressed or uncompressed)
  • Files that require an End User License Agreement (EULA)
  • Confidential information
  • Personal data you do not want publicly available
  • Another’s personally identifiable information
  • Copyrighted materials without the permission of the copyright owner

Files and content that do not abide by the Community Terms of Use or Code of Conduct will be removed. Continued non-compliance may result in NetApp Community account restrictions or termination.