How do asserts and other error avoidance checks fit into TDD?

Question

http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd

You are not allowed to write any production code unless it is to make a failing unit test pass.

So where do asserts fit into this? If you followed this rule you'd never write an assert without a unit test. But how would you test for the existence of an assert? The very nature of assert seems to make this idea impossible as the test would explode.

And what about code meant to handle absurd conditions that are difficult or impossible to replicate? Some sort of FUBAR from a system call for example... You need to handle the error but the condition that causes the error is pretty rare and comes from the OS. How do you write a unit test to make that fail?

Or say you're writing a template library and want to do some concept checks before going too deep into template vomit land? I don't know of any framework that supports compile time failure as a test PASS.

I've tried to live by some of these TDD edicts, but at some point I've always run into walls I can't get around. These are some.

I guess I should say that I'm talking about assert in the C/C++ sense in which a program, built without NDEBUG, will fault out if the assertion fails. Not some exception throwing mechanism that's called assert. Didn't even realize someone would use that name for throwing exceptions.

Answer 1

It looks to me like you're experiencing cognitive dissonance, trying to believe two contradictory ideas and accept both as valid. The way to resolve it is to understand that one (or possibly both) must be incorrect, and find out which it is. In this case, the problem is that those edicts are based on a false premise, which Uncle Bob repeats several times a few lines further down:

However, think about what would happen if you walked in a room full of people working this way. Pick any random person at any random time. A minute ago, all their code worked.

Let me repeat that: A minute ago all their code worked! And it doesn't matter who you pick, and it doesn't matter when you pick. A minute ago all their code worked!

That's the shining promise of TDD: test everything, make it so all your tests pass, and all your code will work.

Problem is, that's a blatant falsehood.

Test everything, make it so all your tests pass, and all your tests will pass, nothing more, nothing less. That doesn't mean anything particularly useful; it only means that none of the error conditions that you thought to test for exist in the codebase. (But if you thought to test for them, then you were paying enough attention to that possibility to write the code carefully enough to get it right in the first place, so that's less helpful than it might be.)

It doesn't mean that any error you didn't think of is not present in the codebase. It also doesn't mean that your tests--which are also code written by you--are bug-free. (Take that concept to its logical conclusion and you end up caught in infinite recursion. It's tests all the way down.)

To give an example, there's an open-source scripting library that I use whose author boasts of over 90% unit test coverage and 100% coverage in all core functionality. But the issue tracker is almost up to 300 bugs now and they keep coming. I think I found five from the first few days of using it in real-world tasks. (To his credit, the author got them fixed very quickly, and it's a good-quality library overall. But that doesn't change the fact that his "100%" unit tests didn't find these issues, which showed up almost immediately under actual usage.)

The other major problem is that as you go on,

every hour you are producing several tests. Every day dozens of tests. Every month hundreds of tests. Over the course of a year you will write thousands of tests.

...and then your requirements change. You have to implement a new feature, or change an existing one, and then 10% of your unit tests break, and you need to manually go over all of them to discern which ones are broken because you made a mistake, and which are broken because the tests themselves are no longer testing for correct behavior. And 10% of thousands of tests is a lot of unnecessary extra work. (Especially if you're doing it 1 test at a time, as the three edicts demand!)

When you think of it, this makes unit testing a lot like global variables, or several other bad design "patterns": it may seem to be helpful and save you some time and effort, but you don't notice the disastrous costs until your project becomes big enough that their overall effect is disastrous, and by that time it's too late.

It is now two decades since it was pointed out that program testing may convincingly demonstrate the presence of bugs, but can never demonstrate their absence. After quoting this well-publicized remark devoutly, the software engineer returns to the order of the day and continues to refine his testing strategies, just like the alchemist of yore, who continued to refine his chrysocosmic purifications.

-- Edsger W. Djikstra. (Written in 1988, so it's now closer to 4.5 decades.)

Answer 2

So where do asserts fit into this?

Test Your Code's Behavior

If you need to test that an assert fires than do so. Pass in the variable that makes the assert throw and exception and in your test catch the exception to assert it was thrown.

And what about code meant to handle absurd conditions that are difficult or impossible to replicate?

You have a few choices.

Don't Test

You have the option of not testing. Testing provides a measure of risk mitigation. You can choose not to test, but realize you increase you risk.

Use Mocks to Facilitate Replication

You can almost always mock out some underlying call to throw the appropriate exception or error condition which can then be tested against.

Indirect Testing

Usually you want to test a condition directly. Sometimes it's too difficult or expensive. In some cases you can mitigate some of the risk by testing for the conditions indirectly. This would include testing for initial states, side effects, or other indicators of the condition.

Exceptions to the Rule

Some cases will always be very difficult to test directly. Testing for a deadlock, cases of resource starvation, threading issues can sometimes be to difficult.

Answer 3

Not sure why you say that "the test would explode". In this example I am testing an assertion:

public int methodUnderTest() {
    int i = resultOfACalculationThatShouldNeverBeZero();
    assert (i != 0);
    return 5 / i;
}

private int resultOfACalculationThatShouldNeverBeZero() {
    return 0;
}

@Test(expected = AssertionError.class)
public void shouldAssertBeforeDividingByZero() {
    methodUnderTest();
}

In a real world scenario the resultOfACalculationThatShouldNeverBeZero would have to be mocked somehow.

Having said that, I don't think assertions (at least in Java) are too useful. It's more like a crutch when debugging, to validate your own assumptions, but probably indicates an overly complex method. One point of the assertion is to document the intent. A unit test does that as well.

Having said all this, I would take what uncle Bob says with a big grain of salt. I have read Clean Code and Agile Software Development and those books contain lots of very very bad code (breaking almost every one of his own rules) and examples where he actually makes the "fixed" code worse than the original..

Answer 4

I use tests and asserts for two different purposes.

I use tests primarily to communicate the desired behavior of my program, and secondarily to validate said behavior. I use asserts primarily to communicate assumptions with potential pitfalls, and secondarily to validate said assumptions.

Neither are silver bullets, and in my experience both function better when viewed as a tool for communicating intent to other programmers, as I feel this is what they're good at. As others have said, don't get caught up in the dogma. Figure out what value the tools offer you and your team, and add them to your process as necessary.