Separating Integration Tests from Unit Tests Using Maven Failsafe & JUnit @Category

Why Unit Tests Should Run Separately From Integration Tests

TDD at the Unit Testing level is fairly straight-forward, since classes in unit testing either do not have complex dependencies, or you mock-out the dependencies with a mocking framework (ex. Mockito). However, TDD quickly becomes difficult when we get to Integration Testing. Integration Testing is basically testing a component with some or all of its dependencies instead of mocking them all out. Examples are tests the cut across multiple layers, tests that read or write to a database or file system, tests that require a Servlet container or EJB container to be up, tests that involve network communication, web services, etc.

Integration Tests tend to be fragile and/or slow. Examples:

• A test that talks to a DB might fail not because the logic in the code was wrong, but because the DB was down, the URL/username/password to the DB was changed, or there was wrong data in the DB.
• Tests that read or write to disk are slow, and each time you run a test the file or database needs to be reset with the proper data or content.
• Packaging and deploying to a container is slow.
• Tests that make network calls may fail not because the logic in the code is wrong, but because the network resource is unavailable, or there's problems with the network itself.

These hassles tend to discourage developers from running tests frequently. When tests are run few and far between, developers end up writing a lot of code before they catch errors. Therefore, when tests are run infrequently, productivity goes down because errors are harder to find and fix after a lot of code has been written, and there's an increased risk of quality issues. Also, when running tests are a hassle, developers are discouraged from writing enough tests.

It therefore makes sense to run unit tests separately from integration tests. Unit tests run completely in memory with no external dependencies, so even for large projects they should all run in just a few seconds, and should run robustly each time since they depend only on the logic of the code under test. Developers are therefore encouraged to run all their unit tests with every small change.

Using Maven Failsafe and JUnit @Category to Separate Integration Tests

There's more than one way to separate integration tests. By default, Failsafe picks up any class with a suffix "IT" or "ITCase", or prefixed with "IT". However, some testing frameworks also require suffixes or prefixes, which makes using that approach cumbersome. Another approach is to place integration tests in a separate source directory. I've chosen to use JUnit @Category since I'm also using Concordion, which requires a suffix in its test classes.

The rest of this article just documents how I implemented the advice from the 2012 article by John Doble entitled, "Unit and Integration Tests With Maven and JUnit Categories". You can find my source code here.

Creating the JUnit Category

Creating a JUnit Category is just simply creating an empty interface. Really, that's it! See below:

package com.orangeandbronze.test;

public interface IntegrationTest {

}

Now, I can apply this "marker interface" as a Category to my integration tests - in the example below, to SectionDaoTest.

import org.junit.experimental.categories.Category;
import com.orangeandbronze.test.IntegrationTest;

@Category(IntegrationTest.class)
public class SectionDaoTest extends DaoTest {
    ...
}

Adding the Surefire and Failsafe Plugins

Now to add the Surefire and Failsafe plugins. I need to exclude all tests marked by IntegrationTest in Surefire (which runs unit tests), and include all the tests marked by IntegrationTest in Failsafe (which runs integration tests). Also, I had to include "**/*.java" or the tests don't run, I don't know why.

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.18.1</version>
  <configuration>
    <excludedGroups>com.orangeandbronze.test.IntegrationTest</excludedGroups>
  </configuration>
</plugin>
<plugin>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.18.1</version>
  <configuration>
    <includes>
      <include>**/*.java</include>
    </includes>
    <groups>com.orangeandbronze.test.IntegrationTest</groups>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>integration-test</goal>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions> 
</plugin>

Running the Tests

So now, when I run mvn test only the unit tests get run, whereas when I run mvn integration-test or mvn verify (I usually run mvn verify), not only do the unit test run, but my project gets packaged and then the integration tests run.

In a real project, each developer would run all the unit tests after just a few changes, dozens of times a day, while he would run the integration tests less frequently, but at least once a day. The CI server would also run both unit and integration tests during its builds.