Sometimes it may be desirable or required in order to comply with regulations to link unit tests back to requirements or risks. In that case you’d need to generate a unit test report listing test name, execution date/time, test result, and requirement/risk IDs. To show you what I mean here’s a standard JUnit HTML test report augmented with extra information:

Idea

The idea is actually quite simple. Use custom annotations on test classes/methods to reference requirements and risks. Then have a custom JUnit RunListener process those annotations to make the information available in the report.

In Java code an excerpt from the sample above would look as follows

@Requirement(references = "[NCA999-555]")
public class PersistenceEditControllerTest {
  @Test
  @Requirement(references = {"[NCA999-666]", "[NCA999-365]" })
  @Risk(references = "R-4711")
  public void testSetupForm() {

Implementation

What you need for this recipe are four ingredients:

1. @Risk and @Requirement annotations for test classes/methods
2. JUnit RunListener implementation
3. Two types of “persisters” which save the annotation data for later use in the JUnit report
4. XSLT style sheet which renders requirement/risk references when the XML report is transformed to HTML

The integration of the ingredients is done with Maven and its Surefire Plugin.

Check out my project on GitHub for all the source code presented here. If you’re stuck please either leave comments here or create an issue on GitHub.

Annotations

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface Requirement {

  String[] references();
}

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface Risk {

  String[] references();
}

Source: Requirement.java, Risk.java

RunListener

public class ReferencesAnnotationsRunListener extends RunListener {

  // default output folder
  private static final String SUREFIRE_REPORTS_FOLDER = "target/surefire-reports";

  // key of the 1st map is the test class name, key of the 2nd map the test method name
  private final Map<String, Map<String, References>> references = new HashMap<>();

  private final String outputFolder;

  /**
   * Constructor that defines the output folder to be relative to the current dir at
   * "target/surefire-reports".
   */
  public ReferencesAnnotationsRunListener() {
    this(SUREFIRE_REPORTS_FOLDER);
  }

  /**
   * Constructor.
   * 
   * @param outputFolder output folder to be relative to the current dir
   */
  public ReferencesAnnotationsRunListener(String outputFolder) {
    this.outputFolder = outputFolder;
  }

  @Override
  public void testRunFinished(Result result) {
    if (!references.isEmpty()) {
      new SurefireXmlReferencesPersister(outputFolder).persist(references);
      new JsonReferencesPersister(outputFolder).persist(references);
    }
  }

  @Override
  public void testStarted(Description description) {
    addAllRequirements(description);
    addAllRisks(description);
  }

  private void addAllRequirements(Description description) {
    Class<?> testClass = description.getTestClass();
    String methodName = description.getMethodName();

    Requirement classRequirementAnnotation = testClass.getAnnotation(Requirement.class);
    Requirement methodRequirementAnnotation = description.getAnnotation(Requirement.class);
    if (methodRequirementAnnotation != null) {
      addRequirements(testClass.getName(), methodName, methodRequirementAnnotation.references());
    }
    if (classRequirementAnnotation != null) {
      addRequirements(testClass.getName(), methodName, classRequirementAnnotation.references());
    }
  }

  private void addAllRisks(Description description) {
    Class<?> testClass = description.getTestClass();
    String methodName = description.getMethodName();

    Risk classRiskAnnotation = testClass.getAnnotation(Risk.class);
    Risk methodRiskAnnotation = description.getAnnotation(Risk.class);
    if (methodRiskAnnotation != null) {
      addRisks(testClass.getName(), methodName, methodRiskAnnotation.references());
    }
    if (classRiskAnnotation != null) {
      addRisks(testClass.getName(), methodName, classRiskAnnotation.references());
    }
  }

  private void addRequirements(String className, String methodName, String[] references) {
    References methodReferences = getMethodReferences(className, methodName);
    Collections.addAll(methodReferences.getRequirements(), references);
  }

  private void addRisks(String className, String methodName, String[] references) {
    References methodReferences = getMethodReferences(className, methodName);
    Collections.addAll(methodReferences.getRisks(), references);
  }

  private References getMethodReferences(String className, String methodName) {
    Map<String, References> classReferences = getClassReferences(className);
    References methodReferences = classReferences.get(methodName);
    if (methodReferences == null) {
      methodReferences = new References();
      classReferences.put(methodName, methodReferences);
    }
    return methodReferences;
  }

  private Map<String, References> getClassReferences(String className) {
    Map<String, References> classReferences = references.get(className);
    if (classReferences == null) {
      classReferences = new HashMap<>();
      references.put(className, classReferences);
    }
    return classReferences;
  }
}

/**
 * Collection of references related to the supported annotations.
 */
@JsonAutoDetect
class References {

  @JsonInclude(value = Include.NON_EMPTY)
  @JsonProperty
  private final List<String> requirements = new ArrayList<>();

  @JsonProperty
  @JsonInclude(value = Include.NON_EMPTY)
  private final List<String> risks = new ArrayList<>();

  public List<String> getRequirements() {
    return requirements;
  }

  public List<String> getRisks() {
    return risks;
  }
}

Source: ReferencesAnnotationsRunListener.java, References.java

Persisters

The information collected by the RunListener is persisted in two ways to allow for flexible post-processing:

• self-contained additional JSON file target/surefire-reports/references.json
• augmented Surefire XML file target/surefire-reports/TEST-<class-name>.xml

In the standard Surefire file each <testcase> element will have two extra attributes like so

<testcase name="..." classname="..." time="0.001" requirements="[NCA999-666],[NCA999-365],[NCA999-555]" risks="R-4711" />

Both persister implementations implement the same interface

/**
 * Persists the references (from annotations) picked up by the run listener.
 */
public interface ReferencesPersister {

  /**
   * Persists the references picked up by the run listener.
   * 
   * @param references organized top-down with the two map keys being class name and method name
   */
  void persist(Map<String, Map<String, References>> references);
}

Source: ReferencesPersister.java

Surefire XML persister

This implementation depends on JDOM.

public final class SurefireXmlReferencesPersister implements ReferencesPersister {

  private final SAXBuilder jdomBuilder = new SAXBuilder();
  private final XPathFactory xPathFactory = XPathFactory.instance();
  private final XMLOutputter outputter = new XMLOutputter();
  private final String outputFolder;

  /**
   * Constructor.
   * 
   * @param outputFolder output folder
   */
  public SurefireXmlReferencesPersister(String outputFolder) {
    this.outputFolder = outputFolder;
    outputter.setFormat(Format.getPrettyFormat());
  }

  @Override
  public void persist(Map<String, Map<String, References>> references) {
    for (Entry<String, Map<String, References>> referencesPerClassEntry : references.entrySet()) {
      String className = referencesPerClassEntry.getKey();

      // parse the XML file, find the relevant <testcase> elements, add attributes to it, save the
      // XML file
      try {
        File file = getSurefireXmlFileFor(className);
        Document document = jdomBuilder.build(file);
        for (Entry<String, References> referencesPerMethodEntry : referencesPerClassEntry.getValue()
            .entrySet()) {
          addReferencesToDocument(document, referencesPerMethodEntry);
        }
        save(document, file);
      } catch (Exception e) {
        throw new RuntimeException("Failed to persist references to Surefire XML files.", e);
      }
    }
  }

  private void addReferencesToDocument(Document document,
      Entry<String, References> referencesPerMethodEntry) {
    Element testcase = getTestcaseElementFromDocument(document, referencesPerMethodEntry.getKey());
    References references = referencesPerMethodEntry.getValue();

    if (!references.getRequirements().isEmpty()) {
      testcase.setAttribute("requirements", collectionToString(references.getRequirements()));
    }

    if (!references.getRisks().isEmpty()) {
      testcase.setAttribute("risks", collectionToString(references.getRisks()));
    }
  }

  private String collectionToString(Collection<String> collection) {
    return Joiner.on(',').join(collection);
  }

  private Element getTestcaseElementFromDocument(Document document, String testcaseNaame) {
    XPathExpression<Element> xPathExpression = xPathFactory.compile(
        getTestcaseXPath(testcaseNaame), Filters.element());

    Element testCase = xPathExpression.evaluate(document).get(0);
    return testCase;
  }

  private String getTestcaseXPath(String testcaseNaame) {
    return "//testcase[@name='" + testcaseNaame + "']";
  }

  private void save(Document document, File file) throws IOException {
    try (FileOutputStream fos = new FileOutputStream(file)) {
      outputter.output(document, fos);
    }
  }

  private File getSurefireXmlFileFor(String className) throws IOException {
    return new File(outputFolder, getSurefireXmlFileNameFor(className)).getCanonicalFile();
  }

  private String getSurefireXmlFileNameFor(String className) {
    return "TEST-" + className + ".xml";
  }
}

Source: SurefireXmlReferencesPersister.java

JSON persister

This implementation depends on Jackson.

public class JsonReferencesPersister implements ReferencesPersister {

  private static final String OUTPUT_FILE_NAME = "references.json";
  private final String outputFolder;

  /**
   * Constructor.
   * 
   * @param outputFolder output folder
   */
  public JsonReferencesPersister(String outputFolder) {
    this.outputFolder = outputFolder;
  }

  @Override
  public void persist(Map<String, Map<String, References>> references) {
    try {
      File outputFile = new File(outputFolder, OUTPUT_FILE_NAME).getCanonicalFile();
      new ObjectMapper().writerWithDefaultPrettyPrinter().writeValue(outputFile, references);
    } catch (Exception e) {
      throw new RuntimeException("Failed to persist references to JSON.", e);
    }
  }
}

Source: JsonReferencesPersister.java

Maven configuration

There a basically two things you need to add to your POM:

• configure the Surefire Plugin to use your RunListener
• use the Maven Antrun Plugin to trigger ant-junitreport which converts the Surefire XML to HTML using XSLT

  <surefire.dir>${project.build.directory}/surefire-reports</surefire.dir>
</properties>

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.18.1</version>
      <configuration>
        <properties>
          <property>
            <name>listener</name>
            <value>ReferencesAnnotationsRunListener</value>
          </property>
        </properties>
      </configuration>
    </plugin>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-antrun-plugin</artifactId>
      <version>1.8</version>
      <executions>
        <execution>
          <id>test-reports</id>
          <phase>prepare-package</phase>
          <goals>
            <goal>run</goal>
          </goals>
          <configuration>
            <target>
              <junitreport todir="${surefire.dir}">
                <fileset dir="${surefire.dir}">
                  <include name="TEST-*.xml" />
                </fileset>
                <report format="noframes" todir="${surefire.dir}" styledir="${where-you-keep-the-xslt-stylesheet.dir}" />
              </junitreport>
            </target>
          </configuration>
        </execution>
      </executions>
      <dependencies>
        <dependency>
          <groupId>org.apache.ant</groupId>
          <artifactId>ant-junit</artifactId>
          <version>1.9.3</version>
        </dependency>
      </dependencies>
    </plugin>
  </plugins>
</build>

Source: pom.xml

The first caveat here is that the XSLT style sheet cannot be loaded from a JAR file but needs to be available through a “regular” file path (line 36). I keep mine in src/main/resources if its a Maven project, Maven will copy it to target/classes during the build process.

The second caveat is that one needs a custom style sheet that includes the extra attributes on the <testcase> element in the final HTML. To get you started you may use mine of course (line 271 has an embedded GIF using data URIs).