Today I wrote "tests" for the testing code. Was it the right thing? Is it a smell?

Question

My application has big integration with the database. Classes that use the database are very crucial for the system, so I write small unit tests for classes that I call Repository. The reason behind the unit-testing database operations are for example verification of caching or correct order of query execution (to avoid foreign key constraints etc). Plus, I write these tests before the actual code since it allows me to work faster.

For unit testing these operations, I use an in-memory database (H2Database - Java). However, since the schema of the database is quite big, creating the schema in the in-memory database takes about ~4 seconds. For hundreds of unit tests such these, this will make my unit-test suite quite slow.

In order to make it faster, instead of erasing completely the in-memory database between tests, I just truncate all the tables from it and not re-create the schema from scratch. This runs quite faster. Truncation takes about 5-10ms and the database is statically shared across all unit tests. This way, 1 test takes 4 seconds. 100 tests take 5 seconds.

But, (to me) that means that this in-memory database/connection is an important part of the test suite. If this does not work correct, I might end up with wrong feedback from my actual tests. So, in the end, I wrote some unit tests that verify the behavior of the test-database. For example, that all tables are truncated every time you call TestInMemoryDb.create().

The fact I wrote "tests" for the test code, is it a smell? Is it a wrong decision to do? I know that since it gives me confidence, it should be alright. But... is it? Or it is unnecessary thing to do?

There is always the option to create between the tests only the parts of the schema I need. But this comes with one disadvantage. All the CREATE TABLE statements in my unit tests are reducing readability and tests are full of noise. Having the schema as a whole in an sql file and then running it, is more convenient and easier to maintain instead of breaking it into pieces here and there.

Answer 1

I disagree that you have written a test for your tests. Consider the reason why you write tests to begin with: to verify non-trivial behavior. What you describe seems like an integration test, even if the code being tested is not used in application code.

Clearly TestInMemoryDb.create() has some complexity. Furthermore, it is a crucial part of the test infrastructure. A failure in this single method could potentially cause hours of debugging. Despite that behavior not being used in the application, these are valuable tests.

As the application evolves, the database schema will change. This will impact the procedure that "cleans" the database and hopefully results in test failures. The assumption is this would save time.

What you did is make a judgement call about one piece of code. It might be a different situation if you decide to write tests for test code as a general matter of practice, regardless of complexity. While, generally, you don't write tests for test code, there is always an exception to the rule. This appears to be a good example of why you should not blindly follow guidelines.

Answer 2

I am sure most major testing tools (like all the ones named with "xyz"-Unit, replace "xyz" by your favorite letter from the alphabet) have unit tests. These are tools complex enough to deserve such tests, and they are written by people who value unit testing. For example, I just checked NUnit and JUnit, they are Open Source, hence you can find their unit tests easily in the source tree.

So when you implement your own tools for your test suite, there is nothing special in writing unit tests for your tools, quite the opposite. This only sounds strange because of the wording: the tests you wrote are not for testing "the tests", they are testing your test tool, which is not the same. Hence don't overthink it.

Answer 3

I can suggest another way to look at it: by making sure your test DB operates how you expect you're not testing your tests. You're testing your test framework.

Off-the-shelf test frameworks have their own tests. But you've implemented bespoke test functionality according to your specific needs, so you should verify it works how you expect.

Answer 4

I think you are heading down a rabbit hole here.

On the face of it you have written some code that "cleans up" databases, and writing a test for that code makes sense. Build that code and use the binary in your other tests without retesting it like you would any other third party library.

However, I think the common way of testing Data Access Layer code these days is to spin up a verison of the real database in a container and run your tests against that. This illiminates and problems caused by discrepencies between an in memory database and the real version.

When it comes to clearing out tables between tests, I would ask you "do you expect your code to run on non-empty databases?" if so, then the tests should pass without clearing the db down.

Using a real db doesn't fix the problem of "clearing my db takes ages" but it does fix the problem of "I am writing a bunch of code and infrastructure which is only ever used in my tests, may be faulty and requires its own test suite"

Answer 5

There's "test infrastrcture" and "test code". You don't want to write tests for "test code" in any circumstances, but you do need to write tests for the "test infrastrcture" if it becomes complicated. These tests can perhaps be running a dummy test case, aka smoke test.