ONTAP Discussions

how to generate 50k files of 75mb each to create a test enviroment.

Jruggero

Does anyone know If there is an easy way to generate 50K files (please note: not 1 million as originally thought)  that are 75MB each, (so that we get the full 3.5TB of data) which is approximately what our suspect process was deleting, I would love to hear suggestions on how to create those files quickly.

 

We are presently running script on an NFS client  to copy /dev/random into a file to create a 75MB filesize, that takes 11 sec/file (so over 6 days for the 50K files)  if we copy that randomly generated file to a new filename, it  take 1.5secs (so still 12+ hrs) to generate 50K files.

2 REPLIES 2

mbeattie

Hi,

 

You can use the "System.IO.File" class to create files easily. Here is an example script:

 

Source Code:

 

Param(
   [Parameter(Position=0,
      Mandatory=$True,
      ValueFromPipeLine=$True,
      ValueFromPipeLineByPropertyName=$True)]
   [String]$UNCPath,
   [Parameter(Position=1,
      Mandatory=$True,
      ValueFromPipeLine=$True,
      ValueFromPipeLineByPropertyName=$True)]
   [Int]$FileCount,
   [Parameter(Position=2,
      Mandatory=$True,
      ValueFromPipeLine=$True,
      ValueFromPipeLineByPropertyName=$True)]
   [Int]$FileSizeMB
)
#'------------------------------------------------------------------------------
Function New-File{
   Param(
      [CmdletBinding()]
      [Parameter(Position=0,
         Mandatory=$True,
         ValueFromPipeLine=$True,
         ValueFromPipeLineByPropertyName=$True)]
      [String]$FileSpec,
         [Parameter(Position=1,
         Mandatory=$True,
         ValueFromPipeLine=$True,
         ValueFromPipeLineByPropertyName=$True)]
      [Double]$FileSize
   )
   [Bool]$result = $True
   Try{
      $file = [System.IO.File]::Create("$FileSpec")
      $file.SetLength($FileSize)
      $file.Close()
   }Catch{
      [Bool]$result = $False
   }
   Return $result;
}
#'------------------------------------------------------------------------------
#'Create random files names of the specified size in the share.
#'------------------------------------------------------------------------------
For($i=1; $i -le $FileCount; $i++){
   [String]$fileName = [System.IO.Path]::GetRandomFileName()
   If(New-File -FileSpec "$uncPath\$fileName" -FileSize ($FileSizeMB * "1MB")){
      Write-Host $("Created file ""$uncPath\$fileName"" of size """ + $FileSizeMB + "MB""")
   }
}
#'------------------------------------------------------------------------------

Example output:

 

PS C:\Scripts\PowerShell\Projects\CreateFiles> .\CreateFiles.ps1 -UNCPath "\\vserver1\vol1$" -FileCount 10 -FileSize 75
Created file "\\vserver1\vol1$\vtiefg0t.lus" of size "75MB"
Created file "\\vserver1\vol1$\1a34gcbk.mew" of size "75MB"
Created file "\\vserver1\vol1$\qorh4d3u.1aj" of size "75MB"
Created file "\\vserver1\vol1$\aqa2wsmn.m5z" of size "75MB"
Created file "\\vserver1\vol1$\cz4yqyjx.nwa" of size "75MB"
Created file "\\vserver1\vol1$\xfvztnrq.3rl" of size "75MB"
Created file "\\vserver1\vol1$\5tcj2qst.i50" of size "75MB"
Created file "\\vserver1\vol1$\olsiqr5t.kcv" of size "75MB"
Created file "\\vserver1\vol1$\bglkpwxx.qqq" of size "75MB"
Created file "\\vserver1\vol1$\jajmi00s.b2k" of size "75MB"

This is useful for appearance of test data but it depends what you want to test...the files are essentially empty so they will dedupe\compress to almost nothing although i'm sure it would be easy enough to have the script append random characters to the files to pad them out.

 

Hope that helps

 

/Matt

 

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

MKELLYSCSC

Try using /dev/urandom instead of /dev/random.

e.g dd if=/dev/urandom of=<random-file-name> ...

 

See https://en.wikipedia.org/wiki//dev/random for the difference between random and urandom.

 

Hope this helps.

Announcements
Register for Insight 2021 Digital

INSIGHT 2021 Digital: Meet the Specialists 2

On October 20-22, gear up for a fully digital, totally immersive virtual experience with a downright legendary lineup of world-renowned specialists. Tune in for visionary conversations, solution deep dives, technical sessions and more.

NetApp on Discord Image

We're on Discord, are you?

Live Chat, Watch Parties, and More!

Explore Banner

Meet Explore, NetApp’s digital sales platform

Engage digitally throughout the sales process, from product discovery to configuration, and handle all your post-purchase needs.

NetApp Insights to Action
I2A Banner
Public