Property-based testing isn’t new. It’s quite common in the functional programming world. Anyway, there is a big chance that as a C# programmer you have never used it before. I heard about it only because I used to work with an awesome F# developer a few years ago (cheers Tomek!). Luckily for us, .NET world is bigger than only C#.
Getting back to the topic, I will explain a few things in this article:
- how to write a property-based test
- how is it different from the usual unit test
- when it makes sense to use it
What is a property?
When looking at the code we can define some properties.
It can be clear, readable, concise, etc.
That’s not what we are looking for when doing property-based testing.
What we need is a trait that can be verified.
Let’s get back to basic math. Properties describe every mathematical function. Let’s look at addition.
- It’s commutative because 1+2 is the same as 2+1.
- It’s associative because (3+2) + 1 equals 3 + (2+1).
- And finally, it has an identity property in the form of number 0. Adding 0 to any number doesn’t change its value.
Of course, we can describe functions and classes by properties as well. And that’s the idea behind property-based testing.
After we find and define those characteristics, we can make some assertions about them.
Right now, you are wondering how to define the properties of your code?
To picture that we will use a vending machine simulator. First, we are going to find attributes defining our functions. Next, we will test those attributes using Property-Based Testing.
Vending Machine simulator
We will start by defining our business needs.
How the usual vending machine behaves?
- It accepts coins
- It returns coins if we didn’t buy anything
- It gives us a specific product if the value of inserted coins is at least as high as the price of the product
- It returns change if the product value is lower than the value of inserted coins
- It doesn’t allow you to buy the product if the value of inserted coins is too low.
Of course, a common vending machine has more functions but, in our case, these few are enough.
To test all cases, we need 4 tests.
- If returning money works
- If one can’t buy a product when the value of inserted coins is too low
- If one can buy a product when inserted the exact value of a product
- If one can buy a product and get a change
We will only focus on the first and last case. It’s should be enough for you to be able to cover remaining cases on your own.
Inserting and returning coins
We expect that our machine will always return the same amount of coins that we inserted. Writing a unit test for that case is straightforward. The simplest example can look like this:
public void ReturnsAllInsertedCoins(string insertedCoins, string expected)
{
var sut = new VendingMachine();
sut.InsertMoney(insertedCoins);
var returnedCoins = sut.Return();
returnedCoins.Should().Be(expected);
}
That would be the usual approach. Pass input data to our function and confirm the outcome.
But I’m sure you can see the flaw with this test. First, we must pass some specific data and know the expected result. We also must validate it. What if there is a need to pass thousands of example values? Are we going to create them by hand?
Of course, there is no need to do that. To help us, we will use the FsCheck library which is written purely in F#. It makes it usable in the whole .NET ecosystem. On the downside, it’s not always easy to use with C# but that’s a story for another time.
To automatically generate data, we need Arbitraries from the above-mention tool. Add xUnit to this equation and we can see the power of property-based testing.
Let’s define property describing the return of money.
It’s straightforward – you return the same coins that one inserted.
The algorithm for such a test looks like this:
- Insert some random coins consisting of values 0.25$, 0.5$ and 1$
- Press return
- Check if the same coins that one inserted were returned.
We are now ready to write our first property-based test.
Let’s start with it.
[Property(Arbitrary = new[] {typeof(CoinArbitraries)})]
public Property ReturnInsertedMoney(string coins)
{
var sut = new VendingMachine();
Func<bool> property = () =>
{
sut.InsertMoney(coins);
return sut.Return() == coins;
};
return property.ToProperty();
}
Here I owe you some explanation.
The [Property] attribute declares a data generator that we are going to use. For now, it’s a random list of coins. Generated values are passed as a parameter into the function.
Inside the Func delegate, lies the definition of our property.
We expect that our Return() function returns the same coins value that one inserted into the machine.
In the last line, we convert our function to property and let FsCheck do the work.
Let’s now look at the data generator, that provides our test with input:
private static readonly decimal[] AllowedCoins = {1m, 0.50m, 0.25m};
public static Arbitrary<string> CoinGenerator()
{
return Gen.Elements(AllowedCoins)
.ListOf()
.Select(x => string.Join(", ", x))
.ToArbitrary();
}
What do we have here?
Using Gen.Elements method from the FsCheck library we generate collection with random coins. Then, using the ListOf() function, we generate a list of random length. By default, it will create 100 lists with different values. Next, after formatting the result, it’s passed to our ReturnInsertedMoney test as a parameter. And that’s all! We have a fully functional test.
Next stop – getting a product with correct change
Now the most important feature. It would be great to get a bag of chips for free but sadly, that’s not how the world works. Also, that would make our vending machine owners furious. To make them happy, we must test returning products with change functionality.
Our algorithm looks like this:
- Generate a random length list of allowed coins
- Pick only these collections that sum of values is higher than the value of the product
- Calculate the expected change
- Return list of coins and change
We can do it like so:
private static readonly decimal[] AllowedCoins = {1m, 0.5m, 0.25m};
public static Arbitrary<(string,string)> MoreThan1Dollar()
{
return Gen.Elements(AllowedCoins)
.ListOf()
. Where(x => x.Sum() > 1.0m)
. Select(x =>
(string.Join(", ", x), (x.Sum() - 1.0m).ToString()))
.ToArbitrary();
}
You can spot some similarities to the previous generator. The biggest difference is that this time, we must pick only specific collections. This makes it more complicated.
So, what’s going on here?
First, the same as previously, we generate a collection of coins (lines 4 and 5). But this time, we must make sure that the value of coins is higher than the price of our product. In this case, it’s 1$. We do that in line 6, with a simple Where clause. Additionally, we must calculate the change for that specific amount of coins. That’s why we need code from line 8, in which we format the coins collection and return it in conjunction with correct change value.
Enough code for now. More tests and Vending Machine Simulator is available at my Github.
Conclusion
First, how is it different from traditional unit testing?
As we’ve seen so far, property-based testing requires a different approach. Instead of checking some specific cases, like we usually do, we let some external library create such cases randomly. The only thing we must do is provide some high-level rules that our code should follow. Then, such data will quickly verify our idea. That’s another great characteristic of this approach. We get a lot of different input values for free. Because of that, when running tests long enough, we will find some examples that our code didn’t expect. And voila! We found a bug. This is impossible to do with the conventional approach.
Ultimately, when property-based testing makes sense? It’s perfect when our code needs to support numerous data, which creation follows a pattern. As in our example with returning inserted coins. It would require a lot of work to define data by hand. But as you noticed, it was easy to do using generators.
I hope I’ve convinced you that property-based testing has a place in day-to-day development. It can replace some parts of our unit tests with a more robust and elegant code. Although it requires a different approach and change of mindset it’s very rewarding. No matter which .NET part is your homeland it’s worth trying it out by yourself!
If you’re interested in .NET technologies and work in an interesting project, apply for a position of .NET Developer at Aspire!