Implementing Long Running Business Processes (a.k.a. Sagas) Using Azure Durable Functions — Shipping
In the previous post, I looked into modeling & implementing Buyer’s Remorse long running process. In this post, I want to look into implementing a shipping start process. Let’s assume the requirement is:
After order accepted & customer billed for an order, kick off a shipping process.
In this case we’re dealing with an orchestration. We need to wait until both events happened before kicking off the shipping process. This is oversimplified version. In a real world case there are many other aspects that would need to be considered. What happens when 1 event is received but we don’t get the second event in a timely fashion. What happens when a failure event (such as order billing failed) is received. In my opinion these questions should be addressed to business stakeholders & I want to keep this simple so I won’t look into handling them.
Let us first look into implementation using an orchestration durable function and challenges it presents.
Durable Orchestration
So we define 2 HTTP triggered functions to represent handlers for OrderAccepted & OrderBilled events. They are defined as HTTP triggered with /GET endpoints for ease of testing (triggering via browser). Idea is the following — for both events, once event is received, we first check if orchestration with a given id exists. If no we kick off an orchestration, if yes we just raise an event which orchestration would wait for.
Now, this implementation imposes a challenge if both functions run concurrently (not likely, but it is possible). In the Microsoft documentation on concurrent execution it notes:
There is a potential race condition in this sample. If two instances of HttpStartSingle execute concurrently, both function calls will report success, but only one orchestration instance will actually start. Depending on your requirements, this may have undesirable side effects. For this reason, it is important to ensure that no two requests can execute this trigger function concurrently.
Projecting this to our sample, if both functions execute at the same time, we can have a situation when both try to start a new orchestration instance. Only one instance will be started, but none of them will raise an event, hence the orchestration won’t finish successfully. Now, we could implement a distributed lock and ensure only one of the functions will create an orchestration, but it is getting complicated.
Let’s see if there is another way.
Durable Entities
What if we think of shipping orchestrator as a durable entity. It keeps the shipping state (whether OrderAccepted & OrderBilled events received) and publishes a message to kick off shipping process once both events received. Let’s look into how it can be implemented.
Microsoft documentation on durable entities states:
To prevent conflicts, all operations on a single entity are guaranteed to execute serially, that is, one after another.
So the concurrency challenge we had before is not a concern. I hope the code is self-explanatory. Entity keeps track of OrderAccepted & OrderBilled boolean flags. Once both of them are true, message is published to a service bus queue.
Looking into event handlers, all they do now is forward calls to corresponding entity methods.
I updated github repository to include Shipping implementation.
In the next post I’ll look into implementing a discount calculation based on the sum of the orders for the last X days. There is a great talk showing how this kind of a requirement can be modeled, I encourage you to watch it first.
