Maven Doesn't Execute All JUnit Tests by Default

Comments

We as developers write many unit tests. The CI pipeline will run all tests with every build. We trust that, if the status is green, everything should be okay.

But what if the unit tests should have failed? I recently learned that Maven does not execute all tests by default.

Consider this simple class:

public class MyTests {

    @Test
    public void firstTest() {
        assertEquals(2, 1 + 1);
    }

    public static class NestedTests {

        @Test
        public void secondTest() {
            assertEquals(3, 1 + 1);
        }
    }
}

We see that firstTest should succeed and secondTest should fail.

Run Nested Tests

Let's see what happen when we run all tests of our project. We will compare the IDE, Gradle and Maven.

First we run all tests within the IDE:

/images/nested-tests-idea.png

Now let's convert this project to Gradle and run the tests:

> Task :test FAILED

demo.MyTests$NestedTests > secondTest FAILED
    java.lang.AssertionError at MyTests.java:18

2 tests completed, 1 failed

And finally we run this as Maven build:

[INFO] --- maven-surefire-plugin:2.22.0:test (default-test) @ surefire-nested-tests-demo ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running demo.MyTests
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.025 s - in demo.MyTests
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

What kind of magic is this? No test failures with Maven, the build was successful!

Maven Surefire Plugin

Maven uses the Surefire Plugin to execute tests during the build. This plugin has several parameters to configure the lookup and the execution of tests. (Documentation)

Include and Exclude Patterns

The include and exclude patterns are used to determine if a class should be considered as a test class.

The default include patterns are **/Test*, **/*Test, **/*Tests and **/*TestCase. The default exclude pattern is **/*$*.

The default exclude pattern is the reason why Maven doesn't execute secondTest. The path of the class file demo/MyTests$NestedTests matches the pattern **/*$*. Therefore, all tests in NestedTests get excluded.

Why does this default exclude pattern exist? I don't know, maybe for performance reasons? But I could trace it back to the very first commit of Surefire.

Overwrite the Standard Configuration

Let's remove the default exclude pattern in our pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.0</version>
    <configuration>
        <excludes>
            <exclude/>
        </excludes>
    </configuration>
</plugin>

When we run Maven again, the build fails:

[INFO] --- maven-surefire-plugin:2.22.0:test (default-test) @ surefire-nested-tests-demo ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running demo.MyTests$NestedTests
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.035 s <<< FAILURE! - in demo.MyTests$NestedTests
[ERROR] secondTest(demo.MyTests$NestedTests) Time elapsed: 0.005 s <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<2>
    at demo.MyTests$NestedTests.secondTest(MyTests.java:18)

[INFO] Running demo.MyTests
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in demo.MyTests
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] MyTests$NestedTests.secondTest:18 expected:<3> but was:<2>
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Conclusion

Usually, there is no need to organize tests in nested classes. But if you do so, make sure all tests get executed both in your IDE and with your build tool.

The executable examples are available in my GitHub repo.

Comments