Designing unit tests for a stateful system

Question

Background

Test Driven Development was popularized after I already finished school and in the industry. I am trying to learn it, but some major things still escape me. TDD proponents say lots of things like (hereafter referred to as the "single assertion principle" or SAP):

For some time I've been thinking about how TDD tests can be as simple, as expressive, and as elegant as possible. This article explores a bit about what it's like to make tests as simple and decomposed as possible: aiming for a single assertion in each test.

Source: http://www.artima.com/weblogs/viewpost.jsp?thread=35578

They also say things like this (hereafter referred to as the "private method principle" or PMP):

You generally don't unit test private methods directly. Since they are private, consider them an implementation detail. Nobody is ever going to call one of them and expect it to work a particular way.

You should instead test your public interface. If the methods that call your private methods are working as you expect, you then assume by extension that your private methods are working correctly.

Source: How do you unit test private methods?

Situation

I am trying to test a stateful data processing system. The system can do different things for the exact same piece of data given what it's state was prior to receiving that data. Consider a straightforward test that builds up state in the system, then tests the behavior that the given method is intended to test.

• SAP suggests that I should not be testing the "state build up procedure", I should assume that the state is what I expect from the build up code and then test the one state change I'm trying to test

• PMP suggests that I can't skip this "state build up" step and just test the methods that govern that functionality independently.

The result in my actual code has been tests that are bloated, complicated, long, and difficult to write. And if the state transitions change, the tests have to be changed... which would be fine with small, efficient tests but extremely time consuming and confusing with these long bloated tests. How is this normally done?

Answer 1

Perspective:

So let's take a step back and ask what TDD is trying to help us with. TDD is trying to help us determine if our code is correct or not. And by correct, I mean "does the code meet the business requirements?" The selling point is that we know changes will be required in the future, and we want to make sure that our code remains correct after we make those changes.

I bring that perspective up because I think it's easy to get lost in the details and lose sight of what we're trying to achieve.

Principles - SAP:

While I'm not an expert in TDD, I think you're missing part of what Single Assertion Principle (SAP) is trying to teach. SAP can be restated as "test one thing at a time." But TOTAT doesn't roll off the tongue as easily as SAP does.

Testing one thing at a time means that you focus on one case; one path; one boundary condition; one error case; one whatever per test. And the driving idea behind that is you need to know what broke when the test case fails, so you can resolve the issue more quickly. If you test multiple conditions (ie. more than one thing) within a test and the test fails, then you have a lot more work on your hands. You first have to identify which of the multiple cases failed and then figure out why that case failed.

If you test one thing at a time, your search scope is a lot smaller and the defect is identified more quickly. Keep in mind that "test one thing at a time" doesn't necessarily exclude you from looking at more than one process output at a time. For example, when testing a "known good path", I may expect to see a specific, resulting value in foo as well as another value in bar and I may verify that foo != bar as part of my test. The key is to logically group the output checks based upon the case being tested.

Principles - PMP:

Likewise, I think you're missing a bit about what Private Method Principle (PMP) has to teach us. PMP encourages us to treat the system like a black box. For a given input, you should get a given output. You don't care how the black box generates the output. You only care that your outputs align with your inputs.

PMP is really good perspective for looking at the API aspects of your code. It can also help you scope what you have to test. Identify your interface points and verify that they meet the terms of their contracts. You don't need to care about how the behind-the-interface (aka private) methods do their job. You just need to verify they did what they were supposed to do.

---

Applied TDD (for you)

So your situation presents a bit of a wrinkle beyond an ordinary application. Your app's methods are stateful, so their output is contingent upon not only the input but also what's been previously done. I'm sure I should <insert some lecture> here about state being horrible and blah blah blah, but that really doesn't help solve your problem.

I'm going to assume you have some sort of state diagram table that shows the various potential states and what needs to be done in order to trigger a transition. If you don't, you're going to need it as it will help express the business requirements for this system.

The Tests: First, you're going to end up with a set of tests that enact state change. Ideally, you'll have tests that exercise the full range of state changes that can occur but I can see a few scenarios where you may not need to go that full extent.

Next, you need to build tests to validate the data processing. Some of those state tests will be reused when you create the data processing tests. For example, suppose you have a method Foo() that has different outputs based upon an Init and State1 states. You'll want to use your ChangeFooToState1 test as a setup step in order to test the output when "Foo() is in State1".

There's some implications behind that approach that I want to mention. Spoiler, this is where I'll infuriate the purists

First off, you have to accept that you using something as a test in one situation and a setup in another situation. On the one hand, this seems to be a direct violation of SAP. But if you logically frame ChangeFooToState1 as having two purposes then you're still meeting the spirit of what SAP is teaching us. When you need to make sure Foo() changes states, then you use ChangeFooToState1 as a test. And when need to validate "Foo()'s output when in State1" then you're using ChangeFooToState1 as a setup.

The second item is that from a practical point of view, you're not going to want fully randomized unit testing for your system. You ought to run all of the state change tests prior to running the output validation tests. SAP is kind of the guiding principle behind that ordering. To state what should be obvious - you can't use something as setup if it fails as a test.

Putting it together:

Using your state diagram, you'll generate tests to cover the transitions. Again, using your diagram, you generate tests to cover all of the input / output data processing cases driven by state.

If you follow that approach, the bloated, complicated, long, and difficult to write tests should get a little bit easier to manage. In general, they should end up smaller and they should be more concise (ie less complicated). You should notice that the tests are more decoupled or modular as well.

Now, I'm not saying the process will be completely pain free because writing good tests does take some effort. And some of them will still be difficult because you're mapping a second parameter (state) over quite a few of your cases. And as an aside, it should be a little more apparent why a stateless system is a easier to build tests for. But if you adapt this approach for your application, you should find that you are able to prove your application is working correctly.

Answer 2

You would usually abstract away the setup details into functions so you don't have to repeat yourself. That way you only have to change it in one place in the test if the functionality changes.

However, you wouldn't normally want to describe even your setup functions as bloated, complicated, or long. That's a sign that your interface needs refactoring, because if it's difficult for your tests to use, it's difficult for your real code to use as well.

That's often a sign of putting too much into one class. If you have stateful requirements, you need a class that manages the state and nothing else. The classes that support it should be stateless. For your SIP example, parsing a packet should be completely stateless. You can have a class that parses a packet then calls something like sipStateController.receiveInvite() to manage the state transitions, which itself calls other stateless classes to do things like ring the phone.

This makes setting up the unit testing for the state machine class a simple matter of a few method calls. If your setup for state machine unit tests requires crafting packets, you've put too much into that class. Likewise, your packet parser class should be relatively simple to create setup code for, using a mock for the state machine class.

In other words, you can't avoid state altogether, but you can minimize and isolate it.

Answer 3

The core idea of TDD is that, by writing tests first, you end up with a system that is, at the least, easy to test. Hopefully it works, is maintainable, well documented and so on, but if not, well at least it's still easy to test.

So, if you TDD and end up with a system that is hard to test, something has gone wrong. Perhaps some things that are private should be public, because you need them to be for testing. Perhaps you are not working at the right level of abstraction; something as simple as a list is stateful at one level, but a value at another. Or perhaps you are giving too much to weight to advice not applicable in your context, or your problem is just hard. Or, of course, perhaps your design is just bad.

Whatever the cause is, you are probably not going to go back and write your system again to make it more testable with simple test code. So likely the best plan is to use some slightly more fancy test techniques, like:

• Model based testing
• Randomised testing