In this article we will look into writing unit tests for Orchestrator functions using xUnit and Moq. We will write tests for a mildly complex orchestrator with branching logic.
So, unit testing…
Before we start, I have to admin, I for one, am not a very big fan of unit testing, I mean, I strive to always 100% coverage for core domain tests, but I seldom test any infrastructure code, and if we are talking about UI’s I basically only write some minimal sanity checks.
The reason for this approach is that during the years, I came to the conclusion, that not all tests are equal, some provide way more value, and some just create noise and are in constant need of updating. For example, updating a major UI framework version, will usually break all tests. Anyway, I always had the luck of working with QA engineers that were doing Selenium tests, so no biggy :).
Never the less, one thing to keep in mind is that TESTING IS IMPORTANT!
Testing Sagas / Orchestrators
Looking at the process of testing orchestrators, at first glance it looks like one of the most useless things that you could be testing.
At least this is what I initially thought when I started my serverless journey, but then I came to realize that the unit tests of the orchestrators could provide an amazing amount of value.
But let’s not get ahead of ourselves, and let’s start the testing.
So, we will be starting with the following orchestrator logic :
We have an orchestrator that will do the following things:
- Check if we deliver to the specified address
- Get the proper suborchestrator to calculate the shipping cost based on the partners in the region
- Invoke the proper suborchestrator to get the right price
- Send an event to a bus for marketing / sales
I know, it is a bit far fetched, but this rather contrived example has what we need to demo the unit tests, to be more exact, we have branching logic, we have a suborchestrator and we have an activity that does not return anything ( believe me, there will be a problem with this one, but you will see how to handle this).
The code
The orchestrator that we will be writing the unit tests looks like this:
public static class SagaToTestOrchestrator
{
[FunctionName("SagaToTestOrchestrator")]
public static async Task<ShippingPrice> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var input = context.GetInput<SagaContext>();
// activity to check if we ship to the specified continent
if (!await context.CallActivityAsync<bool>("IsContinentSupported", input.Continent))
{
return new ShippingPrice()
{
Shippable = false,
Message = "We aren't able to ship to your location"
};
}
// activity to get proper orchestrator for continent for shipping partner
var supplierOrchestratorToRun = await context.CallActivityAsync<string>("GetSupplierOrchestratorForContinent", input.Continent);
// orchestrator to get the price for the shipping address
var priceForShipment =
await context.CallSubOrchestratorAsync<decimal>($"{supplierOrchestratorToRun}Orchestrator", input);
// activity to publish event for Sales / marketing
await context.CallActivityAsync("PublishCalculatedPriceActivity", (input, priceForShipment));
return new ShippingPrice()
{
Shippable = true,
Price = priceForShipment
};
}
[FunctionName("CourierAOrchestrator")]
public static async Task<decimal> CourierAOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context)
{
return 100;
}
[FunctionName("CourierBOrchestrator")]
public static async Task<decimal> CourierBOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context)
{
return 120;
}
[FunctionName("IsContinentSupported")]
public static async Task<bool> IsContinentSupported([ActivityTrigger] string continent, ILogger log)
{
var supportedContinents = new List<string>
{
"North America", "South America", "Europe",
};
return supportedContinents.Contains(continent);
}
[FunctionName("GetSupplierOrchestratorForContinent")]
public static async Task<string> GetSupplierOrchestratorForContinent([ActivityTrigger] string continent, ILogger log)
{
var courier = "";
switch (continent)
{
case "South America":
case "North America":
courier = "CourierA";
break;
case "Europe":
courier = "CourierB";
break;
}
return courier;
}
[FunctionName("PublishCalculatedPriceActivity")]
public static async Task PublishCalculatedPriceActivity([ActivityTrigger] (SagaContext context, decimal price) input, ILogger log)
{
log.LogInformation($"{input.context.Continent}: {input.price}");
}
[FunctionName("SagaToTestOrchestrator_HttpStart")]
public static async Task<HttpResponseMessage> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "post")]
HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync("SagaToTestOrchestrator", null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
}
public class ShippingPrice
{
public bool Shippable { get; set; }
public decimal Price { get; set; }
public string Message { get; set; }
}
public class SagaContext
{
public string Street { get; set; }
public string City { get; set; }
public string Country { get; set; }
public string Continent { get; set; }
}
}
So, in order to try to keep things as simple as possible, I went with the static approach, and also kept everything related to this orchestrator in one file. If you are interested in how to set-up a more structured Durable Function project you could read this article.
Now, this is not what you came here for, so on to the testing part.
Let the testing begin!
So, unit testing, like most of all things in software development could be done in a myriad of ways. Most of them will amount similar results. Now, here we will show the way described in the official MS Docs, after I’ll show how I usually like to test the orchestrators.
The way of the docs
Although, technically the way described in docs and the way we will end up are quite similar, the idea behind is quite different.
So, after reading the documentation this is the first test functions:
[Fact]
public async Task CalculatePriceForAmerica()
{
// Arrange / Given
var orchContext = new SagaContext
{
Continent = "North America"
};
var context = new Mock<IDurableOrchestrationContext>();
// mock the get input
context.Setup(m =>
m.GetInput<SagaContext>()).Returns(orchContext);
//set-up mocks for activities
context.Setup(m =>
m.CallActivityAsync<bool>("IsContinentSupported", It.IsAny<object>()))
.ReturnsAsync(true);
// set-up mocks for activity
context.Setup(m
=> m.CallActivityAsync<string>("GetSupplierOrchestratorForContinent", It.IsAny<object>()))
.ReturnsAsync("CourierA");
// set-up mocks for suborchstrators
context.Setup(m =>
m.CallSubOrchestratorAsync<decimal>("CourierAOrchestrator", It.IsAny<string>(), It.IsAny<object>()))
.ReturnsAsync(100);
// ACT / When
var price = await SagaToTestOrchestrator.RunOrchestrator(context.Object);
// Assert / Then
Assert.True(price.Shippable);
Assert.Equal(100, price.Price);
}
Now, let’s go through the code for a bit. As you can notice we have the three usual sections specific to AAA or GWT methodologies.
As you can see above, the biggest part of the test is the Arrange part. We need to mock all the activities that will be called in the test. Also, we need to mock the GetInput function of the orchestrator and also the suborchestrators.
After, in the “Act” section, we keep it quite simple, we just invoke the orchestrator and get the result.
The assert part then checks we check that the returned value from the orchestrator has the proper values / state.
Now, this code works, and the test passes, but I think there are parts that we can not test. For example using this approach of testing we could never test functions that for example would send events, since they do not influence in any way the result.
Also, there might be cases where you would have Orchestrators that wouldn’t have any concrete result, such as the ones that watch the service bus, and react in some way and then never return anything.
The “Flow Testing” Way
In order to fix this, I use something that I like to refer to as “Flow Testing”. Basically this way, we don’t really want to assert the end result, since in reality we mock all in inputs and functions, so the chances are quite high that the result is good. Instead we will focus on testing that the proper elements in the “flow” were called the right number of times. If “flow” is unclear, you could read more about this here.
So, let’s see some code, and then we will discuss a bit further.
// V2: The Flow Way
[Fact]
public async Task CalculatePriceForEurope()
{
// Arrange / Given
var orchContext = new SagaContext
{
Continent = "Europe"
};
var context = new Mock<IDurableOrchestrationContext>();
// mock the get input
context.Setup(m =>
m.GetInput<SagaContext>()).Returns(orchContext);
//set-up mocks for activities
context.Setup(m =>
m.CallActivityAsync<bool>("IsContinentSupported", It.IsAny<object>()))
.ReturnsAsync(true);
// set-up mocks for activity
context.Setup(m
=> m.CallActivityAsync<string>("GetSupplierOrchestratorForContinent", It.IsAny<object>()))
.ReturnsAsync("CourierB");
// set-up mocks for suborchstrators
context.Setup(m =>
m.CallSubOrchestratorAsync<decimal>("CourierAOrchestrator", It.IsAny<string>(), It.IsAny<object>()))
.ReturnsAsync(100);
context.Setup(m =>
m.CallSubOrchestratorAsync<decimal>("CourierBOrchestrator", It.IsAny<string>(), It.IsAny<object>()))
.ReturnsAsync(120);
// mock the publish activity
// at the time of writing, there is no way of mocking CallActivityAsync so we need to use the generic version
context.Setup(m =>
m.CallActivityAsync<object>("PublishCalculatedPriceActivity", It.IsAny<object>())
);
// ACT / When
var price = await SagaToTestOrchestrator.RunOrchestrator(context.Object);
// Assert / Then
context.Verify(
m => m.CallActivityAsync<bool>(
"IsContinentSupported",
It.IsAny<object>()),
Times.Once);
context.Verify(
m => m.CallActivityAsync<string>(
"GetSupplierOrchestratorForContinent", It.IsAny<object>()),
Times.Once
);
context.Verify(m =>
m.CallSubOrchestratorAsync<decimal>("CourierAOrchestrator", It.IsAny<string>(), It.IsAny<object>()),
Times.Never);
context.Verify(m =>
m.CallSubOrchestratorAsync<decimal>("CourierBOrchestrator", It.IsAny<string>(), It.IsAny<object>()),
Times.Once);
context.Verify( m =>
m.CallActivityAsync<object>("PublishCalculatedPriceActivity", It.IsAny<object>()),
Times.Once
);
}
So, as you can see, using this approach, we do not validate the the values of the mocks that were returned, which is a bit silly, but we test that given the proper values, the orchestrator flow is behaving how we expect it.
Of course, this isn’t an either or problem, so the two ways could be easily combined and do both, but in my case, my main orchestrator rarely returns anything, it’s there to glue several systems together, and most of the time, it is running of a service bus trigger. So this way is better suited.
Bonus, the parameterized flow way
Well this last part, is something that people usually have mixed feelings about, we will be using the [Theory] and [MemberData] attributes from xUnit, to parametrize our unit tests.
This could be achieved more or less due to the fact that this kind of test are rather repeatable and contain a lot of boiler plate.
Here is how the code looks:
// V3 : Parameterized Flow
[Theory]
[MemberData(nameof(DataSourceForTest))]
public async Task TestUsingTheory(OrchestratorTestParams pTestParams)
{
// Arrange / Given
var orchContext = new SagaContext
{
Continent = pTestParams.Continent
};
var context = new Mock<IDurableOrchestrationContext>();
// mock the get input
context.Setup(m =>
m.GetInput<SagaContext>()).Returns(orchContext);
//set-up mocks for activities
context.Setup(m =>
m.CallActivityAsync<bool>("IsContinentSupported", It.IsAny<object>()))
.ReturnsAsync(pTestParams.IsContinentSupported);
// set-up mocks for activity
context.Setup(m
=> m.CallActivityAsync<string>("GetSupplierOrchestratorForContinent", It.IsAny<object>()))
.ReturnsAsync(pTestParams.SupplierToBeReturnedFromContinentOrchestrator);
// set-up mocks for suborchstrators
context.Setup(m =>
m.CallSubOrchestratorAsync<decimal>("CourierAOrchestrator", It.IsAny<string>(), It.IsAny<object>()))
.ReturnsAsync(pTestParams.ValueForCourierA);
context.Setup(m =>
m.CallSubOrchestratorAsync<decimal>("CourierBOrchestrator", It.IsAny<string>(), It.IsAny<object>()))
.ReturnsAsync(pTestParams.ValueForCourierB);
// mock the publish activity
// at the time of writing, there is no way of mocking CallActivityAsync so we need to use the generic version
context.Setup(m =>
m.CallActivityAsync<object>("PublishCalculatedPriceActivity", It.IsAny<object>())
);
// ACT / When
var price = await SagaToTestOrchestrator.RunOrchestrator(context.Object);
// Assert / Then
context.Verify(
m => m.CallActivityAsync<bool>(
"IsContinentSupported",
It.IsAny<object>()),
pTestParams.IsContinentSupportedCalledTimes);
context.Verify(
m => m.CallActivityAsync<string>(
"GetSupplierOrchestratorForContinent", It.IsAny<object>()),
pTestParams.GetSupplierOrchestratorForContinentCalledTimes
);
context.Verify(m =>
m.CallSubOrchestratorAsync<decimal>("CourierAOrchestrator", It.IsAny<string>(), It.IsAny<object>()),
pTestParams.CourierAOrchestratorCalledTimes);
context.Verify(m =>
m.CallSubOrchestratorAsync<decimal>("CourierBOrchestrator", It.IsAny<string>(), It.IsAny<object>()),
pTestParams.CourierBOrchestratorCalledTimes);
context.Verify( m =>
m.CallActivityAsync<object>("PublishCalculatedPriceActivity", It.IsAny<object>()),
pTestParams.PublishCalculatedPriceActivityCalledTimes
);
}
As you can notice, this is almost identical to the “flow” version, the only difference being the use of parameters that are provided from outside.
In order to not have a gazilion of parameters passed, I created a simple container class that has all the params as properties. This looks like this:
public class OrchestratorTestParams
{
public string Continent { get; set; }
public bool IsContinentSupported { get; set; }
public string SupplierToBeReturnedFromContinentOrchestrator { get; set; }
public decimal ValueForCourierA { get; set; }
public decimal ValueForCourierB { get; set; }
public Times IsContinentSupportedCalledTimes { get; set; }
public Times GetSupplierOrchestratorForContinentCalledTimes { get; set; }
public Times CourierAOrchestratorCalledTimes { get; set; }
public Times CourierBOrchestratorCalledTimes { get; set; }
public Times PublishCalculatedPriceActivityCalledTimes { get; set; }
}
Now if you are not familiar with xUnit’s Theory, it tells xUnit to run the unit test function multiple times for each element in the collection provided. Here is a nice article that I found which explains this in much more detail. In our case we used the member data way of passing the parameters. Speaking of which here is how this this looks.
public static IEnumerable<object[]> DataSourceForTest =>
new List<object[]>
{
new object[]
{
new OrchestratorTestParams
{
Continent = "Europe",
IsContinentSupported = true,
SupplierToBeReturnedFromContinentOrchestrator = "CourierB",
ValueForCourierA = 100,
ValueForCourierB = 120,
IsContinentSupportedCalledTimes = Times.Once(),
GetSupplierOrchestratorForContinentCalledTimes = Times.Once(),
CourierAOrchestratorCalledTimes = Times.Never(),
CourierBOrchestratorCalledTimes = Times.Once(),
PublishCalculatedPriceActivityCalledTimes = Times.Once()
}
},
new object[] {
new OrchestratorTestParams
{
Continent = "North America",
IsContinentSupported = true,
SupplierToBeReturnedFromContinentOrchestrator = "CourierA",
ValueForCourierA = 100,
ValueForCourierB = 120,
IsContinentSupportedCalledTimes = Times.Once(),
GetSupplierOrchestratorForContinentCalledTimes = Times.Once(),
CourierAOrchestratorCalledTimes = Times.Once(),
CourierBOrchestratorCalledTimes = Times.Never(),
PublishCalculatedPriceActivityCalledTimes = Times.Once()
}
},
new object[] {
new OrchestratorTestParams
{
Continent = "Antartica",
IsContinentSupported = false,
SupplierToBeReturnedFromContinentOrchestrator = "CourierA",
ValueForCourierA = 100,
ValueForCourierB = 120,
IsContinentSupportedCalledTimes = Times.Once(),
GetSupplierOrchestratorForContinentCalledTimes = Times.Never(),
CourierAOrchestratorCalledTimes = Times.Never(),
CourierBOrchestratorCalledTimes = Times.Never(),
PublishCalculatedPriceActivityCalledTimes = Times.Never()
}
}
};
Quite nice I might say…
Looking at this, we have 3 scenarios running in this theory, the first and second are the exact same scenarios that we tested earlier, and the third on is testing how the orchestrator behaves if we pass un unsupported orchestrator.
Now, as stated earlier, this is not for everybody, some people like to have it scenario separated, some appreciate this way this works. I, personally use this only in very rare occasions, mostly for very repetitive tests, but I thought that it might be of interest for you.
The End
Well, hopefully you reached this far. Thank you for taking the time for reading this. As usual all the code is also available on github.
In the next article we will go through some ways to get even more value out of the orchestration tests, and also make them less verbose.
If you liked the content, I’d suggest you join the mailing list to get notified when the next article will be published.
Laurentiu's corner 