How to skip tests on error in PHPUnit?

Question

How do I get PHPUnit to skip tests when the dependent one has an error with a data set?

---

Works

If my data provider only has things that cause errors, then it will appropriately skip the dependent tests. Note the Skipped: 1

class DataProviderDependsTest extends PHPUnit_Framework_TestCase
{
    public function getDataProvider(){
      return [
        ['non-existent_file.txt'],
      ];
    }

    /**
     *  @dataProvider getDataProvider
     */
    public function testCanBeDependedOn($data){
      $actual = file_get_contents($data);
      $this->assertSame('expected',$actual);
    }

    /**
     *  @dataProvider getDataProvider
     *  @depends testCanBeDependedOn
     */
    public function testCanDepend($data){
      $this->assertTrue(false);
    }
}

PHPUnit 5.5.0 by Sebastian Bergmann and contributors.

ES                                                                  2 / 2 (100%)

Time: 28 ms, Memory: 4.00MB

There was 1 error:

1) DataProviderDependsTest::testCanBeDependedOn with data set #0 ('non-existent_file.txt')
file_get_contents(non-existent_file.txt): failed to open stream: No such file or directory

/home/xenial/phpunittest/test.php:16

ERRORS!
Tests: 1, Assertions: 0, Errors: 1, Skipped: 1.

---

Does Not Work

However, if I add one piece of good data to the provider, then despite the errors caused by the rest, PHPUnit proceeds to execute all the dependent tests anyway (even the corresponding data sets with errors). It doesn't skip anything. Note the added ['real_file.txt'], to the data provider.

class DataProviderDependsTest extends PHPUnit_Framework_TestCase
{
    public function getDataProvider(){
      return [
        ['real_file.txt'],
        ['non-existent_file.txt'],
      ];
    }

    /**
     *  @dataProvider getDataProvider
     */
    public function testCanBeDependedOn($data){
      $actual = file_get_contents($data);
      $this->assertSame('expected',$actual);
    }

    /**
     *  @dataProvider getDataProvider
     *  @depends testCanBeDependedOn
     */
    public function testCanDepend($data){
      $this->assertTrue(false);
    }
}

PHPUnit 5.5.0 by Sebastian Bergmann and contributors.

.EFF                                                                4 / 4 (100%)

Time: 19 ms, Memory: 4.00MB

There was 1 error:

1) DataProviderDependsTest::testCanBeDependedOn with data set #1 ('non-existent_file.txt')
file_get_contents(non-existent_file.txt): failed to open stream: No such file or directory

/home/xenial/phpunittest/test.php:16

--

There were 2 failures:

1) DataProviderDependsTest::testCanDepend with data set #0 ('real_file.txt')
Failed asserting that false is true.

/home/xenial/phpunittest/test.php:25

2) DataProviderDependsTest::testCanDepend with data set #1 ('non-existent_file.txt')
Failed asserting that false is true.

/home/xenial/phpunittest/test.php:25

ERRORS!
Tests: 4, Assertions: 3, Errors: 1, Failures: 2.

PHPUnit doesn't skip @depends tests on error when using @dataProvider

From their docs:

Note

When a test depends on a test that uses data providers, the depending test will be executed when the test it depends upon is successful for at least one data set.

I would like to skip some tests all together if any part of the provided data in the depended test causes an error. Is there any way to workaround this limitation?

---

You can fork these files for quick tests if you want, or just clone:

git clone https://github.com/admonkey/phpunittest.git

Answer 1

Maybe this is the behavior you expect:

<?php

class DataProviderDependsTest extends PHPUnit_Framework_TestCase
{
    protected static $failed = false;

    public function getDataProvider() {
        return [
            ['real_file.txt'],
            ['non-existent_file.txt'],
        ];
    }

    /**
     * @dataProvider getDataProvider
     */
    public function testCanBeDependedOn($data) {
        try {
            $actual = file_get_contents($data);
            self::assertSame('expected', $actual);
        } catch(Exception $e) {
            self::$failed = true;
            throw $e;
        }
    }

    /**
     * @dataProvider getDataProvider
     * @depends testCanBeDependedOn
     */
    public function testCanDepend($data) {
        if (self::$failed) {
            self::markTestSkipped('testCanBeDependedOn failed');
        }
        self::assertTrue(true);
    }
}   

Answer 2

Sorry, this answer does not really address your question as you do need at least one passing record in the data provider for the @depends based test to run. The answer by @iRas looks to do what you might want.

I did not delete this answer as it may still provide some information to others.

The @depends does not do the functionality you expect. It does not mean to not run the test if the other fails.

From the Manual for @depends:

PHPUnit supports the declaration of explicit dependencies between test methods. Such dependencies do not define the order in which the test methods are to be executed but they allow the returning of an instance of the test fixture by a producer and passing it to the dependent consumers. Example 2.2 shows how to use the @depends annotation to express dependencies between test methods.

See the section called “Test Dependencies” for more details.

This is used more to pass data between test functions, not for ensuring that one set of tests do not run if the other test fails.

Sample Test in Documentation

<?php
use PHPUnit\Framework\TestCase;

class StackTest extends TestCase
{
    public function testEmpty()
    {
        $stack = [];
        $this->assertEmpty($stack);

        return $stack;
    }

    /**
     * @depends testEmpty
     */
    public function testPush(array $stack)
    {
        array_push($stack, 'foo');
        $this->assertEquals('foo', $stack[count($stack)-1]);
        $this->assertNotEmpty($stack);

        return $stack;
    }

    /**
     * @depends testPush
     */
    public function testPop(array $stack)
    {
        $this->assertEquals('foo', array_pop($stack));
        $this->assertEmpty($stack);
    }
}
?>

In the tests, the $stack from testEmpty() is returned, and that is passed into testPush. This means that $stack in testPush is defined and is an empty array, as opposed to undefined.

Ideally, your tests should not depend on one or the other passing, and be atomic to indicate if functionality works or not, to help find problems in your code. More dependencies based on a test passing can lead to a lot of errors if one test changes data in a manner that is not expected, then all subsequent tests checking that data would fail, which is not really how you should structure the tests. Test for the good conditions, test for failures, but do not have hard dependencies between the tests for pass and fail.