Sometimes it may be desirable or required in order to comply with regulations to link unit tests back to requirements or risks. In that case you’d need to generate a unit test report listing test name, execution date/time, test result, and requirement/risk IDs. To show you what I mean here’s a standard JUnit HTML test report augmented with extra information:
Idea
The idea is actually quite simple. Use custom annotations on test classes/methods to reference requirements and risks. Then have a custom JUnit RunListener process those annotations to make the information available in the report.
In Java code an excerpt from the sample above would look as follows
1 2 3 4 5 6 | @Requirement(references = "[NCA999-555]") public class PersistenceEditControllerTest { @Test @Requirement(references = {"[NCA999-666]", "[NCA999-365]" }) @Risk(references = "R-4711") public void testSetupForm() { |
Implementation
What you need for this recipe are four ingredients:
@Riskand@Requirementannotations for test classes/methods- JUnit RunListener implementation
- Two types of “persisters” which save the annotation data for later use in the JUnit report
- XSLT style sheet which renders requirement/risk references when the XML report is transformed to HTML
The integration of the ingredients is done with Maven and its Surefire Plugin.
Check out my project on GitHub for all the source code presented here. If you’re stuck please either leave comments here or create an issue on GitHub.
Annotations
|
|
Source: Requirement.java, Risk.java
RunListener
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | public class ReferencesAnnotationsRunListener extends RunListener { // default output folder private static final String SUREFIRE_REPORTS_FOLDER = "target/surefire-reports"; // key of the 1st map is the test class name, key of the 2nd map the test method name private final Map<String, Map<String, References>> references = new HashMap<>(); private final String outputFolder; /** * Constructor that defines the output folder to be relative to the current dir at * "target/surefire-reports". */ public ReferencesAnnotationsRunListener() { this(SUREFIRE_REPORTS_FOLDER); } /** * Constructor. * * @param outputFolder output folder to be relative to the current dir */ public ReferencesAnnotationsRunListener(String outputFolder) { this.outputFolder = outputFolder; } @Override public void testRunFinished(Result result) { if (!references.isEmpty()) { new SurefireXmlReferencesPersister(outputFolder).persist(references); new JsonReferencesPersister(outputFolder).persist(references); } } @Override public void testStarted(Description description) { addAllRequirements(description); addAllRisks(description); } private void addAllRequirements(Description description) { Class<?> testClass = description.getTestClass(); String methodName = description.getMethodName(); Requirement classRequirementAnnotation = testClass.getAnnotation(Requirement.class); Requirement methodRequirementAnnotation = description.getAnnotation(Requirement.class); if (methodRequirementAnnotation != null) { addRequirements(testClass.getName(), methodName, methodRequirementAnnotation.references()); } if (classRequirementAnnotation != null) { addRequirements(testClass.getName(), methodName, classRequirementAnnotation.references()); } } private void addAllRisks(Description description) { Class<?> testClass = description.getTestClass(); String methodName = description.getMethodName(); Risk classRiskAnnotation = testClass.getAnnotation(Risk.class); Risk methodRiskAnnotation = description.getAnnotation(Risk.class); if (methodRiskAnnotation != null) { addRisks(testClass.getName(), methodName, methodRiskAnnotation.references()); } if (classRiskAnnotation != null) { addRisks(testClass.getName(), methodName, classRiskAnnotation.references()); } } private void addRequirements(String className, String methodName, String[] references) { References methodReferences = getMethodReferences(className, methodName); Collections.addAll(methodReferences.getRequirements(), references); } private void addRisks(String className, String methodName, String[] references) { References methodReferences = getMethodReferences(className, methodName); Collections.addAll(methodReferences.getRisks(), references); } private References getMethodReferences(String className, String methodName) { Map<String, References> classReferences = getClassReferences(className); References methodReferences = classReferences.get(methodName); if (methodReferences == null) { methodReferences = new References(); classReferences.put(methodName, methodReferences); } return methodReferences; } private Map<String, References> getClassReferences(String className) { Map<String, References> classReferences = references.get(className); if (classReferences == null) { classReferences = new HashMap<>(); references.put(className, classReferences); } return classReferences; } } |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | /** * Collection of references related to the supported annotations. */ @JsonAutoDetect class References { @JsonInclude(value = Include.NON_EMPTY) @JsonProperty private final List<String> requirements = new ArrayList<>(); @JsonProperty @JsonInclude(value = Include.NON_EMPTY) private final List<String> risks = new ArrayList<>(); public List<String> getRequirements() { return requirements; } public List<String> getRisks() { return risks; } } |
Source: ReferencesAnnotationsRunListener.java, References.java
Persisters
The information collected by the RunListener is persisted in two ways to allow for flexible post-processing:
- self-contained additional JSON file
target/surefire-reports/references.json - augmented Surefire XML file
target/surefire-reports/TEST-<class-name>.xml
In the standard Surefire file each <testcase> element will have two extra attributes like so
1 | <testcase name="..." classname="..." time="0.001" requirements="[NCA999-666],[NCA999-365],[NCA999-555]" risks="R-4711" /> |
Both persister implementations implement the same interface
1 2 3 4 5 6 7 8 9 10 11 12 | /** * Persists the references (from annotations) picked up by the run listener. */ public interface ReferencesPersister { /** * Persists the references picked up by the run listener. * * @param references organized top-down with the two map keys being class name and method name */ void persist(Map<String, Map<String, References>> references); } |
Source: ReferencesPersister.java
Surefire XML persister
This implementation depends on JDOM.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | public final class SurefireXmlReferencesPersister implements ReferencesPersister { private final SAXBuilder jdomBuilder = new SAXBuilder(); private final XPathFactory xPathFactory = XPathFactory.instance(); private final XMLOutputter outputter = new XMLOutputter(); private final String outputFolder; /** * Constructor. * * @param outputFolder output folder */ public SurefireXmlReferencesPersister(String outputFolder) { this.outputFolder = outputFolder; outputter.setFormat(Format.getPrettyFormat()); } @Override public void persist(Map<String, Map<String, References>> references) { for (Entry<String, Map<String, References>> referencesPerClassEntry : references.entrySet()) { String className = referencesPerClassEntry.getKey(); // parse the XML file, find the relevant <testcase> elements, add attributes to it, save the // XML file try { File file = getSurefireXmlFileFor(className); Document document = jdomBuilder.build(file); for (Entry<String, References> referencesPerMethodEntry : referencesPerClassEntry.getValue() .entrySet()) { addReferencesToDocument(document, referencesPerMethodEntry); } save(document, file); } catch (Exception e) { throw new RuntimeException("Failed to persist references to Surefire XML files.", e); } } } private void addReferencesToDocument(Document document, Entry<String, References> referencesPerMethodEntry) { Element testcase = getTestcaseElementFromDocument(document, referencesPerMethodEntry.getKey()); References references = referencesPerMethodEntry.getValue(); if (!references.getRequirements().isEmpty()) { testcase.setAttribute("requirements", collectionToString(references.getRequirements())); } if (!references.getRisks().isEmpty()) { testcase.setAttribute("risks", collectionToString(references.getRisks())); } } private String collectionToString(Collection<String> collection) { return Joiner.on(',').join(collection); } private Element getTestcaseElementFromDocument(Document document, String testcaseNaame) { XPathExpression<Element> xPathExpression = xPathFactory.compile( getTestcaseXPath(testcaseNaame), Filters.element()); Element testCase = xPathExpression.evaluate(document).get(0); return testCase; } private String getTestcaseXPath(String testcaseNaame) { return "//testcase[@name='" + testcaseNaame + "']"; } private void save(Document document, File file) throws IOException { try (FileOutputStream fos = new FileOutputStream(file)) { outputter.output(document, fos); } } private File getSurefireXmlFileFor(String className) throws IOException { return new File(outputFolder, getSurefireXmlFileNameFor(className)).getCanonicalFile(); } private String getSurefireXmlFileNameFor(String className) { return "TEST-" + className + ".xml"; } } |
Source: SurefireXmlReferencesPersister.java
JSON persister
This implementation depends on Jackson.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | public class JsonReferencesPersister implements ReferencesPersister { private static final String OUTPUT_FILE_NAME = "references.json"; private final String outputFolder; /** * Constructor. * * @param outputFolder output folder */ public JsonReferencesPersister(String outputFolder) { this.outputFolder = outputFolder; } @Override public void persist(Map<String, Map<String, References>> references) { try { File outputFile = new File(outputFolder, OUTPUT_FILE_NAME).getCanonicalFile(); new ObjectMapper().writerWithDefaultPrettyPrinter().writeValue(outputFile, references); } catch (Exception e) { |
Source: JsonReferencesPersister.java
Maven configuration
There a basically two things you need to add to your POM:
- configure the Surefire Plugin to use your
RunListener - use the Maven Antrun Plugin to trigger ant-junitreport which converts the Surefire XML to HTML using XSLT
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | <surefire.dir>${project.build.directory}/surefire-reports</surefire.dir> </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.18.1</version> <configuration> <properties> <property> <name>listener</name> <value>ReferencesAnnotationsRunListener</value> </property> </properties> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-antrun-plugin</artifactId> <version>1.8</version> <executions> <execution> <id>test-reports</id> <phase>prepare-package</phase> <goals> <goal>run</goal> </goals> <configuration> <target> <junitreport todir="${surefire.dir}"> <fileset dir="${surefire.dir}"> <include name="TEST-*.xml" /> </fileset> <report format="noframes" todir="${surefire.dir}" styledir="${where-you-keep-the-xslt-stylesheet.dir}" /> </junitreport> </target> </configuration> </execution> </executions> <dependencies> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-junit</artifactId> <version>1.9.3</version> </dependency> </dependencies> </plugin> </plugins> </build> |
Source: pom.xml
The first caveat here is that the XSLT style sheet cannot be loaded from a JAR file but needs to be available through a “regular” file path (line 36). I keep mine in src/main/resources if its a Maven project, Maven will copy it to target/classes during the build process.
The second caveat is that one needs a custom style sheet that includes the extra attributes on the <testcase> element in the final HTML. To get you started you may use mine of course (line 271 has an embedded GIF using data URIs).
Thanx for the interesting blog about adding custom columns to unit test reports, this is exactly what I was looking for! 🙂 Unfortunately the imports in your code examples are missing, for some classes aren’t easy to create without. Would it be possible to add the imports here or please just send me the classes if possible? Would be quite helpful. Anyway, thanx again for all the ideas! Great work!
Good point, thanks for bringing this to my attention. I uploaded all the code to https://github.com/marcelstoer/junit-references.
I’ll be editing this post in the next few days. Coming back to this after several months made me realize that a few important clarifications are missing.
works fine also with tycho-surefire-plugin!
Unfortunately this does not seem to work with Jupiter :(.