PowerShell Try/Catch and Retry

Question

I have a fairly large powershell scripts with many (20+) functions which perform various actions.

Right now all of the code doesn't really have any error handling or retry functionality. If a particular task/function fails it just fails and continues on.

I would like to improve error handling and implement retries to make it more robust.

I was thinking something similar to this:

$tries = 0
while ($tries -lt 5) {
 try{ 
 # Do Something
 # No retries necessary
 $tries = 5;
 } catch {
 # Report the error
 # Other error handling
 }
 }

The problem is that I have many many steps where I would need to do this.

I don't think it make sense to implement the above code 20 times. That seems really superfluous.

I was thinking about writing an "TryCatch" function with a single parameter that contains the actual function I want to call?

I'm not sure that's the right approach either though. Won't I end up with a script that reads something like:

TryCatch "Function1 Parameter1 Parameter2"
TryCatch "Function2 Parameter1 Parameter2"
TryCatch "Function3 Parameter1 Parameter2"

Is there a better way to do this?

Answer 1

If you frequently need code that retries an action a number of times you could wrap your looped try..catch in a function and pass the command in a scriptblock:

function Retry-Command {
 [CmdletBinding()]
 Param(
 [Parameter(Position=0, Mandatory=$true)]
 [scriptblock]$ScriptBlock,
 [Parameter(Position=1, Mandatory=$false)]
 [int]$Maximum = 5,
 [Parameter(Position=2, Mandatory=$false)]
 [int]$Delay = 100
 )
 Begin {
 $cnt = 0
 }
 Process {
 do {
 $cnt++
 try {
 # If you want messages from the ScriptBlock
 # Invoke-Command -Command $ScriptBlock
 # Otherwise use this command which won't display underlying script messages
 $ScriptBlock.Invoke()
 return
 } catch {
 Write-Error $_.Exception.InnerException.Message -ErrorAction Continue
 Start-Sleep -Milliseconds $Delay
 }
 } while ($cnt -lt $Maximum)
 # Throw an error after $Maximum unsuccessful invocations. Doesn't need
 # a condition, since the function returns upon successful invocation.
 throw 'Execution failed.'
 }
}

Invoke the function like this (default is 5 retries):

Retry-Command -ScriptBlock {
 # do something
}

or like this (if you need a different amount of retries in some cases):

Retry-Command -ScriptBlock {
 # do something
} -Maximum 10

The function could be further improved e.g. by making script termination after $Maximum failed attempts configurable with another parameter, so that you can have have actions that will cause the script to stop when they fail, as well as actions whose failures can be ignored.

• Brad: Tested this out and it seems as though with $ScriptBlock.Invoke() if there's an error its not being caught. With write-host statements I confirmed that retry-command is being called, make it into the do loop, make it into the try loop, but i've intentionally entered data into my scriptblock which would cause an error yet no error is caught
• Brad: My script block looks like this: Invoke-Sqlcmd -Query "DELETE FROM This_Table_Does_NOT_Exist" -ServerInstance mydbserver -Database TestDB. If I run the command from outside my script I do get an error: Invoke-Sqlcmd : Invalid object name 'This_Table_Does_NOT_Exist'.
• Ansgar Wiechers: I tested the code before posting, and variables could be used just fine. My sample code was missing a -ErrorAction Continue for the Write-Error statement, though (so that the code won't terminate at the first error).
• CrazyIvan1974: I recommend a configurable sleep after writing out the exception.
• GeorgesZ: If one wants to see the success and verbose stream of the $ScriptBlock in case of exception, Invoke-Command -Command $ScriptBlock may be better than $ScriptBlock.Invoke(). It seems that $ScriptBlock.Invoke() returns those streams and they don't get displayed when the script block throws an exception. In this case, use Write-Error $_.Exception.Message -ErrorAction Continue to display the exception's message.

Answer 2

I adapted @Victor's answer and added:

• parameter for retries
• ErrorAction set and restore (or else exceptions do not get caught)
• exponential backoff delay (I know the OP didn't ask for this, but I use it)
• got rid of VSCode warnings (i.e. replaced sleep with Start-Sleep)

# [Solution with passing a delegate into a function instead of script block](https://stackoverflow.com/a/47712807/)
function Retry()
{
 param(
 [Parameter(Mandatory=$true)][Action]$action,
 [Parameter(Mandatory=$false)][int]$maxAttempts = 3
 )
 $attempts=1 
 $ErrorActionPreferenceToRestore = $ErrorActionPreference
 $ErrorActionPreference = "Stop"
 do
 {
 try
 {
 $action.Invoke();
 break;
 }
 catch [Exception]
 {
 Write-Host $_.Exception.Message
 }
 # exponential backoff delay
 $attempts++
 if ($attempts -le $maxAttempts) {
 $retryDelaySeconds = [math]::Pow(2, $attempts)
 $retryDelaySeconds = $retryDelaySeconds - 1 # Exponential Backoff Max == (2^n)-1
 Write-Host("Action failed. Waiting " + $retryDelaySeconds + " seconds before attempt " + $attempts + " of " + $maxAttempts + ".")
 Start-Sleep $retryDelaySeconds 
 }
 else {
 $ErrorActionPreference = $ErrorActionPreferenceToRestore
 Write-Error $_.Exception.Message
 }
 } while ($attempts -le $maxAttempts)
 $ErrorActionPreference = $ErrorActionPreferenceToRestore
}
# function MyFunction($inputArg)
# {
# Throw $inputArg
# }
# #Example of a call:
# Retry({MyFunction "Oh no! It happened again!"})
# Retry {MyFunction "Oh no! It happened again!"} -maxAttempts 10

Answer 3

Solution with passing a delegate into a function instead of script block:

function Retry([Action]$action)
{
 $attempts=3 
 $sleepInSeconds=5
 do
 {
 try
 {
 $action.Invoke();
 break;
 }
 catch [Exception]
 {
 Write-Host $_.Exception.Message
 } 
 $attempts--
 if ($attempts -gt 0) { sleep $sleepInSeconds }
 } while ($attempts -gt 0) 
}
function MyFunction($inputArg)
{
 Throw $inputArg
}
#Example of a call:
Retry({MyFunction "Oh no! It happend again!"})

Answer 4

Error handling is always going to add more to your script since it usually has to handle many different things. A Try Catch function would probably work best for what you are describing above if you want to have each function have multiple tries. A custom function would allow you to even set things like a sleep timer between tries by passing in a value each time, or to vary how many tries the function will attempt.

• brendan62269: You may also want to figure out why you need to retry steps 5 times. Anyway, having a trycatch function would also be a good place to create log output and maybe sort that out...
• Brad: Its mostly a precaution. So there are a few error scenarios we're trying to cover: 1) Is this a terminal error (should we exit) 2) Is this a non-terminal error and we should continue but warn about the error 3) Is this a temporary error (will retrying help?). An example may be that we're trying to run an operation against a VM or cloud instance which hasn't fully started yet. Waiting and retrying is a good option in this situation. Although eventually it should give up.

Answer 5

I created the following function for this same purpose. I use it in a bunch of my scripts. It allows for retries on both terminating errors (i.e. thrown exceptions) and non-terminating errors (i.e. Write-Error), provides an optional exponential backoff, and allows specified types of errors to not be retries (e.g. there's no point trying to retry Stop-Service -Name AServiceThatDoesNotExist).

function Invoke-ScriptBlockWithRetries {
 [CmdletBinding(DefaultParameterSetName = 'RetryNonTerminatingErrors')]
 param (
 [Parameter(Mandatory = $true, HelpMessage = "The script block to execute.")]
 [ValidateNotNull()]
 [scriptblock] $ScriptBlock,
 [Parameter(Mandatory = $false, HelpMessage = "The maximum number of times to attempt the script block when it returns an error.")]
 [ValidateRange(1, [int]::MaxValue)]
 [int] $MaxNumberOfAttempts = 5,
 [Parameter(Mandatory = $false, HelpMessage = "The number of milliseconds to wait between retry attempts.")]
 [ValidateRange(1, [int]::MaxValue)]
 [int] $MillisecondsToWaitBetweenAttempts = 3000,
 [Parameter(Mandatory = $false, HelpMessage = "If true, the number of milliseconds to wait between retry attempts will be multiplied by the number of attempts.")]
 [switch] $ExponentialBackoff = $false,
 [Parameter(Mandatory = $false, HelpMessage = "List of error messages that should not be retried. If the error message contains one of these strings, the script block will not be retried.")]
 [ValidateNotNull()]
 [string[]] $ErrorsToNotRetry = @(),
 [Parameter(Mandatory = $false, ParameterSetName = 'IgnoreNonTerminatingErrors', HelpMessage = "If true, only terminating errors (e.g. thrown exceptions) will cause the script block will be retried. By default, non-terminating errors will also trigger the script block to be retried.")]
 [switch] $DoNotRetryNonTerminatingErrors = $false,
 [Parameter(Mandatory = $false, ParameterSetName = 'RetryNonTerminatingErrors', HelpMessage = "If true, any non-terminating errors that occur on the final retry attempt will not be thrown as a terminating error.")]
 [switch] $DoNotThrowNonTerminatingErrors = $false
 )
 [int] $numberOfAttempts = 0
 while ($true) {
 try {
 Invoke-Command -ScriptBlock $ScriptBlock -ErrorVariable nonTerminatingErrors
 if ($nonTerminatingErrors -and (-not $DoNotRetryNonTerminatingErrors)) {
 throw $nonTerminatingErrors
 }
 break # Break out of the while-loop since the command succeeded.
 } catch {
 [bool] $shouldRetry = $true
 $numberOfAttempts++
 [string] $errorMessage = $_.Exception.ToString()
 [string] $errorDetails = $_.ErrorDetails
 Write-Verbose "Attempt number '$numberOfAttempts' of '$MaxNumberOfAttempts' failed.`nError: $errorMessage `nErrorDetails: $errorDetails"
 if ($numberOfAttempts -ge $MaxNumberOfAttempts) {
 $shouldRetry = $false
 }
 if ($shouldRetry) {
 # If the errorMessage contains one of the errors that should not be retried, then do not retry.
 foreach ($errorToNotRetry in $ErrorsToNotRetry) {
 if ($errorMessage -like "*$errorToNotRetry*" -or $errorDetails -like "*$errorToNotRetry*") {
 Write-Verbose "The string '$errorToNotRetry' was found in the error message, so not retrying."
 $shouldRetry = $false
 break # Break out of the foreach-loop since we found a match.
 }
 }
 }
 if (-not $shouldRetry) {
 [bool] $isNonTerminatingError = $_.TargetObject -is [System.Collections.ArrayList]
 if ($isNonTerminatingError -and $DoNotThrowNonTerminatingErrors) {
 break # Just break out of the while-loop since the error was already written to the error stream.
 } else {
 throw # Throw the error so it's obvious one occurred.
 }
 }
 [int] $millisecondsToWait = $MillisecondsToWaitBetweenAttempts
 if ($ExponentialBackoff) {
 $millisecondsToWait = $MillisecondsToWaitBetweenAttempts * $numberOfAttempts
 }
 Write-Verbose "Waiting '$millisecondsToWait' milliseconds before next attempt."
 Start-Sleep -Milliseconds $millisecondsToWait
 }
 }
}

And here is an example of using it:

Invoke-ScriptBlockWithRetries { Stop-Service -Name "SomeService" } -ErrorsToNotRetry 'Cannot find any service with service name'

Here's another example of using it for a web request:

[scriptblock] $exampleThatReturnsData = {
 Invoke-WebRequest -Uri 'https://google.com'
}
$result = Invoke-ScriptBlockWithRetries -ScriptBlock $exampleThatReturnsData -MaxNumberOfAttempts 3
if ($result.StatusCode -eq 200) {
 Write-Output "Success"
}

You can read more about building the function, see more examples, and get the latest code here.