Is there a way to turn off 'Stop On Failure/Error' for a specific test in PHPUnit?

Question

I am developing an API documentation system, and want to dynamically check that each command has documentation attached. The easiest way to do this is dynamically loop through each command and check for existing documentation to match it.

My code looks like this:

public function testMissingDocs()
{
    foreach ($aCommands as $sKey => $aOptions)
    {
        $this->assertNotNull($oDocs->get($sKey));
    }
}

The problem with this is the StopOnFailure/Error feature of PHPUnit which stops the test after the first assertion fails. I understand the reasons for this functionality and I want to keep it on for the majority of my test cases, but for dynamic assertions/tests it makes things a bit hard.

Is there a way to disable it on a per-test basis so I can check each command in this test?

Answer 1

You can use a data provider to split the single test into as many tests as you have commands.

/**
 * @dataProvider getDocsForAllCommands
 */
public function testEveryCommandHasDocs($sKey)
{
    $this->assertNotNull($oDocs->get($sKey));
}

public function getKeysForAllCommands()
{
    return array_keys($aCommands);
}

Answer 2

If the documentation for a particular class or method is missing, that would represent a problem with that class, not with the method to retrieve the documentation.

Although it is probably easier to combine all of the documentation into a single test, that does not follow unit testing best practices (and hence is why the PHPUnit framework is working against you rather than for you).

I would suggest one of two approaches to rectify the issue:

1. Refactor your tests so that each class and/or method has its own documentation check (there are a few ways you can run multiple tests in one execution).
2. Use a tool such as PHP_CodeSniffer instead, as lack of documentation could be considered a convention – rather than a functional – defect.