test the return value of a method that triggers an error with PHPUnit

Question

This question is specific to using PHPUnit.

PHPUnit automatically converts php errors to exceptions. Is there a way to test the return value of a method that happens to trigger a php error (either built-in errors or user generated errors via trigger_error)?

Example of code to test:

function load_file ($file)
{
    if (! file_exists($file)) {
        trigger_error("file {$file} does not exist", E_USER_WARNING);
        return false;
    }
    return file_get_contents($file);
}

This is the type of test I want to write:

public function testLoadFile ()
{
    $this->assertFalse(load_file('/some/non-existent/file'));
}

The problem I am having is that the triggered error causes my unit test to fail (as it should). But if I try to catch it, or set an expected exception any code that after the error is triggered never executes so I have no way of testing the return value of the method.

This example doesn't work:

public function testLoadFile ()
{
    $this->setExpectedException('Exception');
    $result = load_file('/some/non-existent/file');

    // code after this point never gets executed

    $this->assertFalse($result);
}

Any ideas how I could achieve this?

Answer 1

There is no way to do this within one unit test. It is possible if you break up testing the return value, and the notice into two different tests.

PHPUnit's error handler catches PHP errors and notices and converts them into Exceptions--which by definition stops program execution. The function you are testing never returns at all. You can, however, temporarily disable the conversion of errors into exceptions, even at runtime.

This is probably easier with an example, so, here's what the two tests should look like:

public function testLoadFileTriggersErrorWhenFileNotFound()
{
    $this->setExpectedException('PHPUnit_Framework_Error_Warning'); // Or whichever exception it is
    $result = load_file('/some/non-existent/file');

}

public function testLoadFileRetunsFalseWhenFileNotFound()
{
    PHPUnit_Framework_Error_Warning::$enabled = FALSE;
    $result = load_file('/some/non-existent/file');

    $this->assertFalse($result);
}

This also has the added bonus of making your tests clearer, cleaner and self documenting.

Re: Comment: That's a great question, and I had no idea until I ran a couple of tests. It looks as if it will not restore the default/original value, at least as of PHPUnit 3.3.17 (the current stable release right now).

So, I would actually amend the above to look like so:

public function testLoadFileRetunsFalseWhenFileNotFound()
{
    $warningEnabledOrig = PHPUnit_Framework_Error_Warning::$enabled;
    PHPUnit_Framework_Error_Warning::$enabled = false;

    $result = load_file('/some/non-existent/file');

    $this->assertFalse($result);

    PHPUnit_Framework_Error_Warning::$enabled = $warningEnabledOrig;
}

Re: Second Comment:

That's not completely true. I'm looking at PHPUnit's error handler, and it works as follows:

• If it is an E_WARNING, use PHPUnit_Framework_Error_Warning as an exception class.
• If it is an E_NOTICE or E_STRICT error, use PHPUnit_Framework_Error_Notice
• Else, use PHPUnit_Framework_Error as the exception class.

So, yes, errors of the E_USER_* are not turned into PHPUnit's *_Warning or *_Notice class, they are still transformed into a generic PHPUnit_Framework_Error exception.

Further Thoughts

While it depends exactly on how the function is used, I'd probably switch to throwing an actual exception instead of triggering an error, if it were me. Yes, this would change the logic flow of the method, and the code that uses the method... right now the execution does not stop when it cannot read a file. But that's up to you to decide whether the requested file not existing is truly exceptional behaviour. I tend to use exceptions way more than errors/warnings/notices, because they are easier to handle, test and work into your application flow. I usually reserve the notices for things like depreciated method calls, etc.

Answer 2

Use a phpunit.xml configuration file and disable the notice/warning/error to Exception conversion. More details in the manual. It's basically something like this:

<phpunit convertErrorsToExceptions="false"
         convertNoticesToExceptions="false"
         convertWarningsToExceptions="false">
</phpunit>

Answer 3

Instead of expecting a generic "Exception", what about expecting a "PHPUnit_Framework_Error" ?

Something like this might do :

/**
 * @expectedException PHPUnit_Framework_Error
 */
public function testFailingInclude()
{
    include 'not_existing_file.php';
}

Which, I suppose, might also be written as :

public function testLoadFile ()
{
    $this->setExpectedException('PHPUnit_Framework_Error');
    $result = load_file('/some/non-existent/file');

    // code after this point never gets executed

    $this->assertFalse($result);
}

For more informations, see Testing PHP Errors
Especially, it says (quoting) :

PHPUnit_Framework_Error_Notice and PHPUnit_Framework_Error_Warning represent PHP notices and warning, respectively.

Looking at the /usr/share/php/PHPUnit/TextUI/TestRunner.php file I have on my system, I see this (line 198 and following) :

if (!$arguments['convertNoticesToExceptions']) {
    PHPUnit_Framework_Error_Notice::$enabled = FALSE;
}

if (!$arguments['convertWarningsToExceptions']) {
    PHPUnit_Framework_Error_Warning::$enabled = FALSE;
}

So maybe you'll have to pass some kind of parameter to activate that behaviour ? But it seems to be enabled by default...

Answer 4

Actually there is a way to test both the return value and the exception thrown (in this case an error converted by PHPUnit).

You just have to do the following:

public function testLoadFileTriggersErrorWhenFileNotFound()
{
    $this->assertFalse(@load_file('/some/non-existent/file'));

    $this->setExpectedException('PHPUnit_Framework_Error_Warning'); // Or whichever exception it is
    load_file('/some/non-existent/file');
}

Notice that to test for the return value you have to use the error suppression operator on the function call (the @ before the function name). This way no exception will be thrown and the execution will continue. You then have to set the expected exception as usual to test the error.

What you cannot do is test multiple exceptions within a unit test.

Answer 5

This answer is a bit late to the party, but anyhow:

You can use Netsilik/BaseTestCase (MIT License) to test directly for triggered Notices/Warnings, without ignoring them or converting them to Exceptions. Because the notices/warnings they are not converted to an Exception, the execution is not halted.

composer require netsilik/base-test-case

Testing for an E_USER_NOTICE:

<?php
namespace Tests;

class MyTestCase extends \Netsilik\Testing\BaseTestCase
{        
    public function test_whenNoticeTriggered_weCanTestForIt()
    {
        $foo = new Foo();
        $foo->bar();

        self::assertErrorTriggered(E_USER_NOTICE, 'The notice message');
    }
}

Hope this helps someone in the future.