(Why) is it important that a unit test not test dependencies?

Question

I understand the value of automated testing and use it wherever the problem is well-specified enough that I can come up with good test cases. I've noticed, though, that some people here and on StackOverflow emphasize testing only a unit, not its dependencies. Here I fail to see the benefit.

Mocking/stubbing to avoid testing dependencies adds complexity to the tests. It adds artificial flexibility/decoupling requirements to your production code to support mocking. (I disagree with anyone who says this promotes good design. Writing extra code, introducing things like dependency injection frameworks, or otherwise adding complexity to your codebase to make things more flexible/pluggable/extensible/decoupled without a real use case is overengineering, not good design.)

Secondly, testing dependencies means that critical low-level code that's used everywhere gets tested with inputs other than those that whoever wrote its tests explicitly thought of. I've found plenty of bugs in low-level functionality by running unit tests on high level functionality without mocking out the low-level functionality it depended on. Ideally these would have been found by the unit tests for the low-level functionality, but missed cases always happen.

What's the other side to this? Is it really important that a unit test doesn't also test its dependencies? If so, why?

I can understand the value of mocking external dependencies like databases, networks, web services, etc. I am referring to internal dependencies, i.e. other classes, static functions, etc. that don't have any direct external dependencies.

Answer 1

It's a matter of definition. A test with dependencies is an integration test, not a unit test. You should also have an integration test suite. The difference is that the integration test suite may be run in a different testing framework and probably not as part of the build because they take longer.

For our product: Our unit tests are run with each build, taking seconds. A subset of our integration tests runs with each check-in, taking 10 minutes. Our full integration suite is run each night, taking 4 hours.

Answer 2

Testing with all the dependencies in place is still important, but it's more in the realm of integration testing as Jeffrey Faust said.

One of the most important aspect of unit testing is to make your tests trustworthy. If you don't trust that a passing test really means things are good and that a failing test really means a problem in the production code, your tests aren't nearly as useful as they can be.

In order to make your tests trustworthy, you have to do a few things, but I'm going to focus on just one for this answer. You have to make sure they are easy to run, so that all of the developers can easily run them before checking code in. "Easy to run" means that your tests run quickly and there's no extensive configuration or setup needed to make them go. Ideally, anyone should be able to check out the latest version of the code, run the tests right away, and see them pass.

Abstracting away dependencies on other things (file system, database, web services, etc.) enables you to avoid requiring configuration and makes you and other developers less susceptible to situations where you're tempted to say "Oh, tests failed because I don't have the network share set up. Oh well. I'll run them later."

If you want to test what you do with some data, your unit tests for that business logic code shouldn't care about how you get that data. Being able to test the core logic of your application without depending on supporting stuff like databases is awesome. If you're not doing it, you're missing out.

P.S. I should add that it's definitely possible to overengineer in the name of testability. Test-driving your application helps mitigate that. But in any event, poor implementations of an approach don't make the approach less valid. Anything can be misused and overdone if one doesn't keep asking "why am I doing this?" while developing.

---

As far as internal dependencies are concerned, things get a bit muddy. The way I like to think about it is that I want to protect my class as much as possible from changing for the wrong reasons. If I have a setup like something like this...

public class MyClass 
{
    private SomeClass someClass;
    public MyClass()
    {
        someClass = new SomeClass();
    }

    // use someClass in some way
}

I generally don't care how SomeClass is created. I just want to use it. If SomeClass changes and now requires parameters to the constructor... that's not my problem. I shouldn't have to change MyClass to accommodate that.

Now, that's just touching on the design part. Far as unit tests are concerned, I also want to protect myself from other classes. If I'm testing MyClass, I like knowing for a fact that there are no external dependencies, that SomeClass didn't at some point introduce a database connection or some other external link.

But an even bigger issue is that I also know that some of my methods' results rely on the output from some method on SomeClass. Without mocking/stubbing out SomeClass, I might have no way to vary that input in demand. If I'm lucky, I might be able to compose my environment inside the test in such a way that it'll trigger the right response from SomeClass, but going that way introduces complexity into my tests and makes them brittle.

Rewriting MyClass to accept an instance of SomeClass in the constructor enables me to create a fake instance of SomeClass that returns the value I want (either via a mocking framework or with a manual mock). I don't generally have to introduce an interface in this case. Whether to do so or not is in many ways a personal choice that may be dictated by your language of choice (e.g. interfaces are more likely in C#, but you definitely wouldn't need one in Ruby).

Answer 3

Aside from the unit vs integration test issue, consider the following.

Class Widget has dependencies on classes Thingamajig and WhatsIt.

A unit test for Widget fails.

In which class lies the problem?

If you answered "fire up the debugger" or "read through the code until I find it", you understand the importance of testing only unit, not the dependencies.

Answer 4

Imagine that programming is like cooking. Then unit testing is the same as making sure your ingredients are fresh, tasty, etc. Whereas integration testing is like making sure your meal is delicious.

Ultimately, making sure that your meal is delicious (or that your system works) is the most important thing, the ultimate goal. But if your ingredients, I mean, units, work, you will have a cheaper way to find out.

Indeed, if you can guarantee that your units/methods work, you are more likely to have a functioning system. I emphasize "more likely", as opposed to "certain". You still need your integration tests, just like you still need someone to taste the meal you cooked and tell you that the end product is good. You will have an easier time getting there with fresh ingredients, that's all.

Answer 5

Testing units by completely isolating them will allow to test ALL possible variations of data and situations this unit can be submitted. Because it is isolated from the rest you can ignore variations that do not have a direct effect on the unit to be tested. this in turn will greatly diminish the complexity of the tests.

Testing with various level of dependencies integrated will allow you to test specific scenarios that might not have been tested in the unit tests.

Both are important. Just doing unit test you will invariably more complex subtle error occurring when integrating components. Just doing integration testing means you are testing a system without the confidence that the individual parts of the machine have not been tested. To think you can achieve better complete test just by doing integration testing is almost impossible as the more components you add the number of entry combination grows VERY fast (think factorial) and creating a test with sufficient coverage becomes very quickly impossible.

So in short, the three levels of "unit testing" I typically use in almost all my projects :

• Unit tests, isolated with mocked or stubbed dependencies to test the hell of a single component. Ideally one would attempt complete coverage.
• Integration tests to test the more subtle errors. Carefully crafted spot checks that would show limit cases and typical conditions. Complete coverage is often not possible, effort focused on what would make the system fail.
• Specification testing to inject various real life data in the system, instrument the data (if possible) and observe the output to ensure it conforms to the specification and business rules.

Dependency injection is one very efficient means to achieve this as it allows to isolate components for unit tests very easily without adding complexity to the system. Your business scenario may not warrant the use of injection mechanism but your testing scenario almost will. This for me is sufficient to make then indispensable. You can use them as well to test the different levels of abstraction independently through partial integration testing.

Answer 6

A lot of good answers. I'd also add a couple of other points:

Unit testing also allows you to test your code when you dependencies don't exist. E.g. you, or your team, have not written the other layers yet, or maybe you are waiting on an interface delivered by another company.

Unit tests also mean that you don't have to have a full environment on your dev machine (e.g. a database, a web server etc). I would strong recommend that all developers do have such an environment, however cut down it is, in order to repro bugs etc. However, if for some reason it is not possible to mimic the production environment then unit testing at least gives yu some level of confidence in your code before it goes in to a larger test system.

Answer 7

What's the other side to this? Is it really important that a unit test doesn't also test its dependencies? If so, why?

Unit. Means singular.

Testing 2 things means that you have the two things and all the functional dependencies.

If you add a 3rd thing, you increase the functional dependencies in the tests beyond linearly. Interconnections among things grow more quickly than the number of things.

There are n(n-1)/2 potential dependencices among n items being tested.

That's the big reason.

Simplicity has value.

Answer 8

(This is a minor answer. Thanks @TimWilliscroft for the hint.)

Faults are easier to localize if:

• Both the unit-under-test and each of its dependencies are tested independently.
• Each test fail if and only if there is a fault in the code covered by the test.

This works nicely on paper. However, as illustrated in OP's description (dependencies are buggy), if the dependencies aren't being tested, it would be hard to pinpoint the location of the fault.

Answer 9

Remember how you first learned to do recursion? My prof said "Assume you have a method that does x" (e.g. solves fibbonacci for any x). "To solve for x you have to call that method for x-1 and x-2". In the same regards, stubbing out dependencies lets you pretend they exist and test that the current unit does what it should do. The assumption is of course that you are testing the dependencies as rigorously.

This is in essence the SRP at work. Focusing on a single responsibility even for your tests, removes the amount of mental juggling you have to do.

Answer 10

About the design aspects: I believe that even small projects benefit from making the code testable. You don't necessarily have to introduce something like Guice (a simple factory class will often do), but seperating the construction process from the programming logic results in

• clearly documenting the dependencies of every class via it's interface (very helpful for people who are new on the team)
• the classes becoming much clearer and easier to maintan (once you put the ugly object graph creation into a seperate class)
• loose coupling (making changes much easier)

Answer 11

Uh ... there are good points on unit and integration testing written in these answers here!

I miss the cost-related and practical views here. That said I see clearly the benefit of very isolated/atomic unit tests (maybe highly independent from each other and with the option to run them in parallel and without any dependencies, like databases, filesystem etc.) and (higher level) integration tests, but ... it also is a matter of costs (time, money, ...) and risks.

So there are other factors, that are much more important (like "what to test") before you think of "how to test" from my experience...

Does my customer (implicitely) pay for the extra amount of writing and maintaining tests? Is a more test-driven approach (write tests first before you write the code) really cost efficient in my environment (code failure risk/cost analyis, people, design specs, set up test environment)? Code is always buggy, but could it be more cost effective to move the testing to the production usage (in the worst case!)?

It also depends a lot on what your code quality (standards) is or the frameworks, IDE, design principles etc. you and your team follow and how experienced they are. A well written, easily understandable, good enough documented (ideally self-documenting), modular, ... code introduces likely less bugs than the opposite. So the real "need", pressure or overall maintenance/bugfix costs/risks for extensive tests may not be high.

Let's take it to the extreme where a co-worker in my team suggested, we have to try to unit test our pure Java EE model layer code with a desired 100% coverage for all classes within and mocking the database. Or the manager that would like the integration tests to be covered with a 100% of all possible real world use cases and web ui workflows because we do not want to risk any use case to fail. But we have a tight budget of about 1 million euros, quite some tight plan to code everything. A customer environment, where potential app bugs would not be a very big danger for humans or companies. Our app will be internally tested with (some) important unit tests, integration tests, key customer tests with designed test plans, a test phase etc. We do not develop an app for some nuclear factory or the pharmaceutical production! (We have only one designated database, easy to test clone for each dev and tightly coupled to the webapp or model layer)

I myself try to write test if possible and develop while unit testing my code. But I often do it from a top-down approach (integration testing) and try to find the good point, where to make the "app layer cut" for the important tests (often in the model-layer). (because it's often a lot about the "layers")

Furthermore unit and integration test code doesn't come without negative impact on time, money, maintenance, coding etc. It is cool and should be applied, but with care and considering the implications when starting or after 5 years of a lot of developed test code.

So I would say it really depends a lot on trust and cost/risk/benefit evaluation ... like in real life where you can't and don't want to run around with a lot or 100% safety mechanisms.

The kid can and should climb up somewhere and may fall down and hurt itself. The car may stop working because I filled in the wrong fuel (invalid input :)). The toast may be burned if the time button stopped working after 3 years. But I don't, never, want to drive on the highway and hold my detached steering wheel in my hands :)

Answer 12

You are correct. In unit tests you should also involve dependencies if they are part of the same code base. Dependencies should only be mocked/stubbed when absolutely necessary, for example, if the dependency is non-deterministic (like current date/time) or an external systems like a web services.

A core principle of unit testing is that a unit test should use the interface of the code-under-test but should not be coupled to the implementation details. If the code under test is entirely rewritten - but the interface and observable behavior remain the same - the tests should not be affected. This is what enables safe refactoring.

So if a unit test tests the behavior of class A, it should not care if class A calls into classes B and C. Refactoring the code to use more or fewer classes should not affect the test, as long as the observable behavior remains the same.

Mocking dependencies unnecessarily just leads to brittle tests and makes refactoring more difficult. It creates more work but produces less value.

That said, mocking/stubbing can be necessary and useful in some particular cases:

• The dependency have non-deterministic behavior. For example, if code depends on the current date or time, a pre-configured "current" data/time should be injected to make the tests reproducible.

• The dependency have side-effects. For example sending mails through mail-gateway. You don't want the test code to send actual mails, so you replace the gateway with a mock which records the mails instead, allowing the test to inspect them afterward. (This kind of mock is sometimes called a spy)

• The dependency is interactive. e.g. prompting a user for input. The UI layer or user input should be mocked to make the test reproducible.

Badly designed code can require more use of mocking. If classes or modules mix business logic with non-deterministic input, you either need to refactor the code to extract the non-deterministic sections, or you need to mock the whole class in order to unit-test code which depends on it. Obviously, this lead to worse test-coverage.

Bottom line: Only mock/stub dependencies when they would prevent the unit test from being fast, reproducible and side-effect free.

---

Footnote about definitions: A unit test does not become an integration test just because it touches more than one class! Integration test refer to testing the integration between subsystems, e.g. client/server, GUI/backend, or integrations with external services. Typically integration tests requires a different approach and tools than unit test, and may be more cumbersome to set up.

In contrast, unit tests only touch a single code base, is written by the developers during development, possibly (but not necessarily) using test-driven development. Unit tests should be isolated and reproducible. Isolated means that the test should not affect any other test running before, after or in parallel. A unit test should interact with a single internal interface and it should test a single behavior of that interface. And crucially, a unit test should test the observable behavior of the interface (typically input/output) but not be coupled to the implementation.

Answer 13

A “unit test” with dependencies can fail if either the unit fails or the dependency fails. So in case of a failure, you have less information what failed.

How much of a problem is that? If you have a dependency that is rock solid, has passed all its test, and hasn’t changed for the last two years, no problem. If you have a dependency that is under active development, and broken in some way or another most of the time, big problem. But if you develop both the unit and the dependency, it’s less of a problem because it shouldn’t be a big problem for you to find when the problem is in the dependency and you should be able to fix it. You need to fix it anyway, because you are testing both the unit and the dependency.

Mocking is definitely of advantage to test your unit’s error handling. If there are very rare error conditions for the dependency that are difficult to reproduce, then mocks can do that easily.