Handling Transient Errors in Durable Functions

One of the things which are certain in the case of non-trivial code running in production are errors. Sooner or later they will happen and they don't even have to result from bugs. Some resources might be down, there can be a network problem, hardware failure, and many other short-lived issues. This is why resiliency is a key aspect of every application. In this post, I want to focus on handling transient errors in a very specific context - Durable Functions.

A Word of Caution - Don’t Use Polly

The fact that Durable Functions provide their own mechanism for automatic retry on failure should be a strong suggestion that it's the way it should be done. But we are creatures of habits and we often bring those habits from other types of applications to serverless ones. One of such habits is using Polly for retries. In the case of Durable Functions (or Azure Functions in general), this can have serious consequences. The time spent at awaits is counted as execution time, this will include delays between retries. Yes, the rules are different in orchestrator functions but you can't perform I/O operations there and those are the ones you will most likely want to retry. So, the built-in automatic retry is the way to go, but sometimes it might be tricky to achieve some scenarios.

Built-in Automatic Retry

To discuss the built-in automatic retry in Durable functions I'm going to use the below activity as an example.

[FunctionName(nameof(MessageOfTheDayActivity))]