We are crafting a .NET framework based CRM web application. In our application, we have many time-consuming asynchronous processes running in the background. For that, we did azure function orchestration using azure storage queues and tables. Soon, we ran into limitations of complexity; as the application started to grow. With the application, orchestration also started to grow. Because of that, we faced significant challenges such as Error Handling, Unmanageable Code.
Thus to overcome those challenges, we started using the Durable Azure Function. This article aims to explain how to achieve Orchestration using a Durable Function, an extension of the Azure Function.
Let's write a simple billing service to sell services based on the subscription model. Following are the steps of the billing service workflow:
- Get the active subscribers list.
- Bill/Charge users monthly.
- Notify users about successful/failed billing charges.
- Generate invoice reports, post the monthly billing finished for all the subscribers.
- In the end, send the invoice report to the client by email.
Implementation Using Azure Function:
The above-listed billing workflow needs to be executed in a sequence. To do that, we wired three function apps one after another. The output of one function will become an input for the next function.
As shown in the above screenshot, the Billing Function App is a Timer Trigger azure function. It will execute every day. The Billing Function app will charge all the active subscribers who have passed the billing date. Based on the charge success/failure response, it pushes a message in email-msg-queue. This will trigger the Email Sending Function App to notify the subscriber about billing. Billing Function App will push the message in invoice-generation-queue to complete the billing. Also, it will trigger the Invoice Generation Function, and it will generate invoice reports. Email Sending Function App will send the invoice reports to the service owner.
Limitations of using Azure Functions:
- Error Handling: In case if Payment Service goes down in the middle of processing. Well, that’s why asynchronous messaging is helpful. On the failure of function execution, the message goes back to the queue. The next execution will pick up this message again for processing. For most event sources like
Azure Storage Queues, the retries happen immediately. Here, implementing an exponential back-off algorithm will be a better idea. At this point, to decide retry timing we must know the history of the previous execution. Hence, retry logic becomes state-full. - Short-Lived:
Azure Functionslive up to a few minutes only. In case if we are having a long list of active subscribers to bill and payment service took a long time to process. Then the function will get a timeout in middle by skipping the function execution. - Flexible sequencing: Here the functions are tightly coupled(hard-defined). In case, we want to assign some add-on services to subscribers. Then, we need to add one more function between the billing and email sending function. Then, it would require the code change of input and output definitions for both functions. Hence functions are not decoupled.
- Fan In/Out: In the above billing service, we need to reconcile the outcomes of all the functions to generate the invoice report. As payment is an asynchronous process, we have to write some custom code to check whether the billing process gets completed for all the customers.
Azure Functionscan't be triggered by two events. In our case, the function is dependent on output from the other two different functions which are running parallelly and we are not sure which one will complete at the end, so we have to write some custom wiring code here to achieve this.
Also, we need to consider if any of the functions fail while the process is running. Then, when re-running the process, we need to make sure it should not affect the storage state. e.g., it should not re-send the email or charge the customers again if once done.
To overcome the above challenges, Azure Durable Orchestration comes into the picture.
Durable Orchestration is like an orchestration in music. The orchestrator ensures that everyone is following the melody, but it's the responsibility of each musician to play their instrument.
An orchestrator function is accountable for starting and tracking a series of functions. Activities are responsible for the execution of a task. Orchestrator is a delegator that delegates work to Stateless Activity Function. It also maintains the big picture and history of the flow. Both the Orchestrator and Activity are the Azure functions.
Let's see how we can put in place the Billing Service using Durable Orchestration.
The following snippet shows the use of orchestration client binding. It starts a new O_ProcessBillingSubscription orchestration function instance from a timer triggered function.
/// <summary>
/// Run every day at 1 AM
/// </summary>
const string TIMER_INFO = "0 0 1 * * *";
[FunctionName("Starter_SubscriptionBilling")]
public async Task BillingSubscriptionDurableOrchestratorAsync(
[TimerTrigger(TIMER_INFO)]TimerInfo myTimer,
[DurableClient] IDurableOrchestrationClient starter)
{
await starter.StartNewAsync("O_ProcessBillingSubscription", null);
}Below is the code snippet of billing process workflow:
[FunctionName("O_ProcessBillingSubscription")]
public async Task ProcessAllBillingSubscriptionAsync(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
//get list of subscription whose BillingDate is passed
var subscriptionlist = await context.CallActivityWithRetryAsync<List<UserSubscriptionModel>>("A_GetSubscriptionListToBill",
new RetryOptions(TimeSpan.FromSeconds(10), 10),null);
var subscriptionTasks = new List<Task<bool>>();
subscriotionlist.ForEach(item =>
{
var paymentResponse= await context.CallActivityWithRetryAsync<PaymentResponseModel>("A_ChargeUserSubscription",
new RetryOptions(TimeSpan.FromSeconds(10), 10), item);
subscriptionTasks.Add(context.CallActivityWithRetryAsync<bool>("A_SendEmail",
new RetryOptions(TimeSpan.FromSeconds(10), 10), paymentResponse));
});
await Task.WhenAll(subscriptionTasks);
var invoiceData= await context.CallActivityWithRetryAsync<InoviceReportData>("A_GenerateInvoice",
new RetryOptions(TimeSpan.FromSeconds(10), 10), subscriotionlist);
await context.CallActivityWithRetryAsync<PaymentResponseModel>("A_SendEmail",
new RetryOptions(TimeSpan.FromSeconds(10), 10), invoiceData);
} a. In the above code, we will first call Activity Function A_GetSubscriptionListToBill. This will give us active subscribers who have passed the billing date.
b. For each subscription in the list, it will call the A_ChargeUserSubscription activity function. A_ChargeUserSubscription calls Payment service API. Based on the API response it sends an email to the users through A_SendEmail activity.
c. A_GenerateInvoice activity will get invoked after processing all billing subscriptions. It will generate an invoice report. The invoice report will be the input for the A_SendEmail activity. It will send an email to the client about the invoice report.
d. Durable functions support retry policy to handle function failures. We can customize it using the CallActivityWithRetryAsync method, and the RetryOptions class. It is used to add the back-off strategy for retrying by setting the Back-off coefficient value in it. In the above code, if any of the activity function fails then it will retry after 10 sec for 10 times. Also, we can use rewind functionality provided by Durable to debug and see the failure.
In case of failure, the orchestrator will not execute from start. It will check the execution history of each activity function. Based on the output of functions it will call the next function. Orchestrator function sleeps when any of the child function gets a call. On completion of child function, it wakes up. Hence, it solves the issue of function timeout. Also, we can add one more step in the workflow without changing i/o definitions of other functions.
We can achieve Fan In/Out using durable orchestration. It fans out the charge billing subscription and email sending activity functions. The durable orchestration will aggregate the results of all the function instances. It will be the input for the Invoice Generation App to generate the invoice report.
Durable functions have provided us the ability to orchestrate our azure functions. It will encourage more people to use Azure Functions to build their compute logic.
Thank you for reading.
References: