how to generate 50k files of 75mb each to create a test enviroment.

[Jruggero]

Does anyone know If there is an easy way to generate 50K files (please note: not 1 million as originally thought)  that are 75MB each, (so that we get the full 3.5TB of data) which is approximately what our suspect process was deleting, I would love to hear suggestions on how to create those files quickly.

We are presently running script on an NFS client  to copy /dev/random into a file to create a 75MB filesize, that takes 11 sec/file (so over 6 days for the 50K files)  if we copy that randomly generated file to a new filename, it  take 1.5secs (so still 12+ hrs) to generate 50K files.

[mbeattie]

Hi,

You can use the "System.IO.File" class to create files easily. Here is an example script:

Source Code:

Param(
   [Parameter(Position=0,
      Mandatory=$True,
      ValueFromPipeLine=$True,
      ValueFromPipeLineByPropertyName=$True)]
   [String]$UNCPath,
   [Parameter(Position=1,
      Mandatory=$True,
      ValueFromPipeLine=$True,
      ValueFromPipeLineByPropertyName=$True)]
   [Int]$FileCount,
   [Parameter(Position=2,
      Mandatory=$True,
      ValueFromPipeLine=$True,
      ValueFromPipeLineByPropertyName=$True)]
   [Int]$FileSizeMB
)
#'------------------------------------------------------------------------------
Function New-File{
   Param(
      [CmdletBinding()]
      [Parameter(Position=0,
         Mandatory=$True,
         ValueFromPipeLine=$True,
         ValueFromPipeLineByPropertyName=$True)]
      [String]$FileSpec,
         [Parameter(Position=1,
         Mandatory=$True,
         ValueFromPipeLine=$True,
         ValueFromPipeLineByPropertyName=$True)]
      [Double]$FileSize
   )
   [Bool]$result = $True
   Try{
      $file = [System.IO.File]::Create("$FileSpec")
      $file.SetLength($FileSize)
      $file.Close()
   }Catch{
      [Bool]$result = $False
   }
   Return $result;
}
#'------------------------------------------------------------------------------
#'Create random files names of the specified size in the share.
#'------------------------------------------------------------------------------
For($i=1; $i -le $FileCount; $i++){
   [String]$fileName = [System.IO.Path]::GetRandomFileName()
   If(New-File -FileSpec "$uncPath\$fileName" -FileSize ($FileSizeMB * "1MB")){
      Write-Host $("Created file ""$uncPath\$fileName"" of size """ + $FileSizeMB + "MB""")
   }
}
#'------------------------------------------------------------------------------

Example output:

PS C:\Scripts\PowerShell\Projects\CreateFiles> .\CreateFiles.ps1 -UNCPath "\\vserver1\vol1$" -FileCount 10 -FileSize 75
Created file "\\vserver1\vol1$\vtiefg0t.lus" of size "75MB"
Created file "\\vserver1\vol1$\1a34gcbk.mew" of size "75MB"
Created file "\\vserver1\vol1$\qorh4d3u.1aj" of size "75MB"
Created file "\\vserver1\vol1$\aqa2wsmn.m5z" of size "75MB"
Created file "\\vserver1\vol1$\cz4yqyjx.nwa" of size "75MB"
Created file "\\vserver1\vol1$\xfvztnrq.3rl" of size "75MB"
Created file "\\vserver1\vol1$\5tcj2qst.i50" of size "75MB"
Created file "\\vserver1\vol1$\olsiqr5t.kcv" of size "75MB"
Created file "\\vserver1\vol1$\bglkpwxx.qqq" of size "75MB"
Created file "\\vserver1\vol1$\jajmi00s.b2k" of size "75MB"

This is useful for appearance of test data but it depends what you want to test...the files are essentially empty so they will dedupe\compress to almost nothing although i'm sure it would be easy enough to have the script append random characters to the files to pad them out.

Hope that helps

/Matt

[MKELLYSCSC]

Try using /dev/urandom instead of /dev/random.

e.g dd if=/dev/urandom of=<random-file-name> ...

See https://en.wikipedia.org/wiki//dev/random for the difference between random and urandom.

Hope this helps.