Get owner of ~5 million files and folders using Powershell and Robocopy

Question

I've written a Powershell script that uses Robocopy to get the full path of all files on a drive and then run it through a foreach loop to get the owner for each file and generate it in a CSV. The code works, but is very slow when I run it against so many items. Can someone help me make it as efficient as possible? I also use "File System Security PowerShell Module 4.2.3" to use Get-NTFSOwner which gets me around the 256 character limit, which is also why I use Robocopy to generate the paths.

Import-Module NTFSSecurity
#Gets initial file and folder list
 $Folderpath = "G:\DEPT"
 $Folders = Get-Childitem $Folderpath
 ForEach ($Folder in $Folders) {Start-Job -ScriptBlock {
 Param($Folder,$Folderpath)
 robocopy "$FolderPath\$Folder" NULL /L /S /NJH /FP /NC /XJ /NS /NJH /NC /NJS /LOG:D:\AutoAssign\Data\RoboLogs\$Folder.txt
 } -ArgumentList $Folder,$Folderpath}
#Wait unitl all jobs are finished
Get-Job | Wait-Job
#Trims out white space and empty lines
 $Folderpath = "D:\AutoAssign\Data\RoboLogs"
 $Files = Get-Childitem $Folderpath
ForEach ($File in $Files) {Start-Job -ScriptBlock {
 Param($File,$Folderpath,$Files)
 $Trim = Get-Content "$Folderpath\$File"
 $Trim.trim() | Where { $_ } | Out-File "$Folderpath\$File"
 } -ArgumentList $File,$Folderpath,$Files }
#Wait unitl all jobs are finished
Get-Job | Wait-Job
#Gets Owner for each file And Create CSV
$Dir = "D:\AutoAssign\Data\RoboLogs"
$Files = Get-ChildItem $Dir
ForEach ($File in $Files){ Start-Job -ScriptBlock {
 Param($Dir,$Files,$File)
$OutputFile = "D:\AutoAssign\Data\Final\$File.csv"
$Results = @()
$Paths = Get-Content $Dir\$File
ForEach ($Path in $Paths){
 $Owner = get-ntfsowner $Path | Select Owner | ft -hidetableheaders | Out-String
 $Owner = $Owner.Trim()
 $Properties = @{
 Path = $Path
 Owner = $Owner
 }
If ($Owner -ne "BUILTIN\Administrators" -and $Owner -ne $null-and $Owner -ne "Domain\Domain Admins" -and $Owner -notlike "S-1*"){
 $Results += New-Object psobject -Property $properties}
 }
$Results | Export-Csv -notypeinformation -Path $OutputFile
} -ArgumentList $Dir,$Files,$File #Ends Initial ForEach
} #Ends Job Script Block
#Wait unitl all jobs are finished
Get-Job | Wait-Job
#Merge all files into one CSV
Get-ChildItem D:\AutoAssign\Data\Final | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv D:\AutoAssign\Data\G.csv -NoTypeInformation -Append
#Delete all original files
Get-ChildItem D:\AutoAssign\Data\RoboLogs | Remove-Item
Get-ChildItem D:\AutoAssign\Data\Final | Remove-Item

• LotPings: What PowerShell Version ? I'd avoid using $Result+= as it rebuilds the whole array each time.
• Josh: Using PowerShell 5.1. Can you recommend what I can substitute in to replace $Results += ?
• Josh: How about this? New-Object psobject -Property $properties | Export-Csv -notypeinformation -Append -Path $OutputFile

Answer 1

Your robocopy command lists /NC and /NJH twice.

$Folders and $Folder contain objects with properties, use these instead of treating them as strings (which implicit casts them as such) or pipe to Select-Object -ExpandProperty FullName Same with $Files and $File

I'd not reuse variable names but define neccessary ones from begin. Proper indenting eases reading/understanding the code IMO better than excess white space.

The following script isn't tested at all, see it as rough template:

## Q:\Test2018円08円12円\cr_201527.ps1
Import-Module NTFSSecurity
$Logs = 'D:\AutoAssign\Data\RoboLogs\'
$Final = 'D:\AutoAssign\Data\Final\'
$Folderpath = "G:\DEPT"
#Gets initial file and folder list
$Folders = Get-Childitem $Folderpath
ForEach ($Folder in $Folders) {
 Start-Job -ScriptBlock { Param($Folder,$Logs)
 robocopy "$($Folder.FullName)" NULL /L /S /NJH /FP /NC /XJ /NS /NJH /NC /NJS /LOG:"$Logs$($Folder.Name).txt"
 } -ArgumentList $Folder,$Logs
}
#Wait until all jobs are finished
Get-Job | Wait-Job
<# functionality integrated into next sction
 #Trims out white space and empty lines
#>
#Gets Owner for each file And Create CSV
ForEach ($File in (Get-ChildItem $Logs)){
 Start-Job -ScriptBlock { Param($Logs,$File,$Final)
 $OutputFile = Join-Path $Final "$($File.Name).csv"
 $Results = ForEach ($Path in ((Get-Content $File.FullName).Trim() -ne '')){
 $Owner = (Get-NtfsOwner $Path).Owner.Trim()
 If ($Owner -ne "BUILTIN\Administrators" -and
 $Owner -ne $null -and
 $Owner -ne "Domain\Domain Admins" -and
 $Owner -notlike "S-1*"){
 [PSCustomObject]@{
 Path = $Path
 Owner = $Owner
 }
 }
 }
 $Results | Export-Csv -NoTypeInformation -Path $OutputFile
 } -ArgumentList $Logs,$File,$Final #Ends Initial ForEach
} #Ends Job Script Block
#Wait unitl all jobs are finished
Get-Job | Wait-Job
#Merge all files into one CSV
Get-ChildItem D:\AutoAssign\Data\Final | Select-Object -ExpandProperty FullName |
 Import-Csv | Export-Csv D:\AutoAssign\Data\G.csv -NoTypeInformation -Append
#Delete all original files
Get-ChildItem D:\AutoAssign\Data\RoboLogs | Remove-Item
Get-ChildItem D:\AutoAssign\Data\Final | Remove-Item