Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
how to generate 50k files of 75mb each to create a test enviroment.
2017-01-10
08:15 PM
3,627 Views
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Does anyone know If there is an easy way to generate 50K files (please note: not 1 million as originally thought) that are 75MB each, (so that we get the full 3.5TB of data) which is approximately what our suspect process was deleting, I would love to hear suggestions on how to create those files quickly.
We are presently running script on an NFS client to copy /dev/random into a file to create a 75MB filesize, that takes 11 sec/file (so over 6 days for the 50K files) if we copy that randomly generated file to a new filename, it take 1.5secs (so still 12+ hrs) to generate 50K files.
2 REPLIES 2
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
You can use the "System.IO.File" class to create files easily. Here is an example script:
Source Code:
Param( [Parameter(Position=0, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [String]$UNCPath, [Parameter(Position=1, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [Int]$FileCount, [Parameter(Position=2, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [Int]$FileSizeMB ) #'------------------------------------------------------------------------------ Function New-File{ Param( [CmdletBinding()] [Parameter(Position=0, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [String]$FileSpec, [Parameter(Position=1, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [Double]$FileSize ) [Bool]$result = $True Try{ $file = [System.IO.File]::Create("$FileSpec") $file.SetLength($FileSize) $file.Close() }Catch{ [Bool]$result = $False } Return $result; } #'------------------------------------------------------------------------------ #'Create random files names of the specified size in the share. #'------------------------------------------------------------------------------ For($i=1; $i -le $FileCount; $i++){ [String]$fileName = [System.IO.Path]::GetRandomFileName() If(New-File -FileSpec "$uncPath\$fileName" -FileSize ($FileSizeMB * "1MB")){ Write-Host $("Created file ""$uncPath\$fileName"" of size """ + $FileSizeMB + "MB""") } } #'------------------------------------------------------------------------------
Example output:
PS C:\Scripts\PowerShell\Projects\CreateFiles> .\CreateFiles.ps1 -UNCPath "\\vserver1\vol1$" -FileCount 10 -FileSize 75 Created file "\\vserver1\vol1$\vtiefg0t.lus" of size "75MB" Created file "\\vserver1\vol1$\1a34gcbk.mew" of size "75MB" Created file "\\vserver1\vol1$\qorh4d3u.1aj" of size "75MB" Created file "\\vserver1\vol1$\aqa2wsmn.m5z" of size "75MB" Created file "\\vserver1\vol1$\cz4yqyjx.nwa" of size "75MB" Created file "\\vserver1\vol1$\xfvztnrq.3rl" of size "75MB" Created file "\\vserver1\vol1$\5tcj2qst.i50" of size "75MB" Created file "\\vserver1\vol1$\olsiqr5t.kcv" of size "75MB" Created file "\\vserver1\vol1$\bglkpwxx.qqq" of size "75MB" Created file "\\vserver1\vol1$\jajmi00s.b2k" of size "75MB"
This is useful for appearance of test data but it depends what you want to test...the files are essentially empty so they will dedupe\compress to almost nothing although i'm sure it would be easy enough to have the script append random characters to the files to pad them out.
Hope that helps
/Matt
If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try using /dev/urandom instead of /dev/random.
e.g dd if=/dev/urandom of=<random-file-name> ...
See https://en.wikipedia.org/wiki//dev/random for the difference between random and urandom.
Hope this helps.