17 Sep '04 - + 24 - 32 Java Code Coverage Tools Evaluation
Over the past week, I've been evaluating various test Code Coverage
tools for our Java application. We have a large Java project
with about 1500 production classes and 550 test classes. My goal
was to
find a tool that integrates well into our Ant test target,
automatically producing html reports at the application, package,
class, and line level without requiring code changes. The tool
can't be too slow. Good
documentation and examples, historical reports, and an IntelliJ IDEA
plugin
were nice-to-have but not necessary. Cost is a
consideration. I looked at ten free tools and one commercial tool
during the evaluation. I found that some projects are dead, some
have some work to do, and some have created excellent tools that should
work for any team.
Introduction
JavaRanch
provides a nice overview of code coverage, explaining the goals of code
coverage and demonstrating a few tools. I'll rely on that and
Google to provide my introduction to the subject. I will say that
most successful tools work by instrumenting the bytecode to track which
code is accessed. When a line of bytecode is executed, the
instrumentation records the event, and those records are later
processed into the reports. The instrumented bytecode is usually
created in a separate directory which needs to be placed in the
classpath for the JUnit task.
The best listing of free Java code coverage tools is at java-source.net.
I considered each of the tools on that page and Clover, a commercial
product. Here are my evaluations, from least suitable to most
suitable.
Dead projects
InsECT
is a dead project. Its Sourceforge activity is 0%, having not
released anything since version 0.91 in 2003. I didn't evaluate it.
JVDMI Code Coverage Analyser
is also a dead project. Its Sourceforge activity is 26%, having
not released anything since version 0.2 in 2002. I didn't
evaluate it.
Interesting approach, but not for us
Jester does not instrument
the byte code to present a report, either. Instead it finds code
that hasn't been tested and literally changes the code. It
changed branching conditions, hard-coded numbers, and String literals
to see which tests break. While this is an interesting idea that
did
really well for Robert Martin and Robert Koss's
bowling scoring example, it is not what we are looking for.
Hansel does not instrument
the byte code to present a report. Instead, it requires the developer to insert a
CoverageDecorator in the suite() method of the test. This
decorator will cause the test to fail if any of the code in the target
class is not executed. While this approach would be very
effective, it would not work for us. Since we have over 500 tests
now, this change would require a lot of work to change the tests, and
the sudden failure of a few thousand test cases would make our current
tests irrelevant. This tool might be effective for a new project,
but not for us.
Closer, but didn't work for us
NoUnit is a promising
project that could work well for some projects, but it didn't quite
meet our needs. The biggest problem is that it doesn't come with
an Ant task. While I did see on forums that people have written
their own Ant tasks, I want that functionality to be built into the
tool. Also, NoUnit reports to the method level, while most other
tools go a step further to check lines within the methods.
Quilt is popular, but I
can't figure out why. Sourceforge activity is high, yet the latest
version is from October 2003. Perhaps the project will improve
significantly soon, but I was not impressed. The documentation was
produced in Maven, so it looked good, but there was very little meat to
it. In fact, I couldn't find a single screenshot of the final report.
More troublesome, I couldn't get the tutorial to work with our
application, and the example that came with the binary distribution did
not work. There are enough good alternatives not to consider this tool.
jcoverage/gpl
is the free version of a product that also has two different commercial
versions. I was very frustrated with the site because it also
didn't show a screen shot or the cost of the non-free versions.
The tutorial that came with the gpl version (the only one I tried) was
minimal, but I thought it had enough to show me how to use the
tool. Unfortunately, Ant couldn't find all of their tasks despite
my following their directions, so I never really got the tool to
work. Since there are better free and commercial application with
much better documentation, I didn't press the issue.
JBlanket is an
academic project that seems to have a similar approach to many other
tools here. The screen shots in the documentation look
great. I really wanted it to work, but there was a failure
somewhere between JBlanket and JDOM. I had to move on.
Quilt, JBlanket, and
jcoverage didn't work for our application and the examples given;
perhaps I would have solved the problems with more stubbornness, but
the poor documentation did not encourage me to think that they would be
better than Emma, which I had already evaluated.
GroboUtils
would be a great product for a smaller product, but it needs much
better performance before we could use it. Instrumenting our
bytecode took almost twenty minutes on my machine, and the
documentation implies that the instrumentation should happen on every
compilation. Also, I got an OutOfMemory error during report
generation. On the other hand, GroboUtils has nice functionality
and examples, and I'm sure it would be a great solution for a smaller
project.
The winners
Emma is a great free
product. It started out as an internal tool, and it has been on
Sourceforge since May 2004. Its instrumentation and report
generation are very fast, taking less than two minutes each where
GroboUtils took over ten minutes for each. The
reports
are very nice, although they don't use pretty green and red bars like
some of the other tools use. There are HTML, XML, and text
reports, and the HTML report allows drill-down to the line. Emma
uses an interesting approach to defining coverage that often results in
lines being only partially covered. For example, lines with
ternary operators with only one branch executed will show as partially
executed, which is very nice. Emma is the only free tool that
worked on our application and satisfied all my requirements. The
examples and documentation were thorough, although I ran into a couple
errors in the basic Ant tutorial. Emma does not provide automated
historical reports or an IntelliJ plugin.
Clover is the best tool of
the bunch. At its price ($2500 for the first five developers,
$100 for each after that), it should be. Its instrumentation and
report
generation are very fast, similar to Emma. The
HTML reports
are even nicer than Emma's, showing nice green and red bars to
demonstrate coverage percentage in addition to the number. The
HTML reports also allow drill-down to the line level, and they are
designed with both frame and no-frame versions that will be familiar to
anyone who knows Javadoc. There are also XML and PDF
reports. Clover also provides an executive report and historical
reports to show a high-level overview of coverage, and how the coverage
has changed over time. Also, Clover integrates completely with
IntelliJ and other IDEs.
The biggest problem I had with Clover may have been an artifact of
evaluation; I couldn't evaluate both the plugin and the Ant version at
the same time because Clover instruments the actual bytecode instead of
a copy, and the two installations of Clover fought over which had
control. I hope that this problem goes away with the full,
purchased version of Clover. Either that, or I hope that I will
find a fix to the problem in a forum somewhere.
Conclusion
Emma and
Clover
were
the best tools. Emma is fast, free, integrates with Ant
well, and produces nice reports. Clover is fast, integrates with
Ant and IntelliJ well, produces even nicer reports, and also has
historical reports. However, it costs $2,500 to get the full set
of features. Both have good documentation.
I do not know
which we will start using; that will depend on getting budgetary
approval for Clover. However, Emma is good enough that we'll be
very happy with it as a free alternative.
Appendix
| Tool | License | sf.net Activity (Sep 14, 2004) | Advantages | Disadvantages |
|---|
| Clover | Various
Pricing | N/A | Great
reports, IDE integration, history reports, nice Ant
integration | Because it instruments the main
bytecode, it takes over; cost
|
| Emma | Common Public License v1.0 | 98.0256% | Very nice reports, fast
| Have to create our own historical reports |
| GroboUtils | MIT
License | 82.2061% | Good
examples and documentation - good for small application
| Instrumentation and report generation are too slow for our large application |
| NoUnit | GPL | 57.1157% | Nice report
| No Ant task; measures method-level coverage exercised by JUnit tests only, not statement and branch coverage |
| Jester | Open License | 56.5148% | Tests fragility of tests | It
doesn't really test coverage
|
| Hansel | BSD License | 44.0677% | Makes the tests themselves fail | Relies on changing all the suite() methods, not on instrumenting the byte code |
| Quilt | Apache
License and Artistic License | 82.3901% | None | Weak
tutorial and documentation; couldn't
get it working |
| jcoverage/gpl | GPL | N/A | None | Couldn't
get it working |
| JBlanket | GPL | N/A | None | Couldn't
get it working |
| JVDMI Code Coverage Analyser | LGPL | 26.703% | None | Inactive |
| InsECT | GPL | 0% | None | Inactive |
Note: Comments were closed on June 9, 2007 due to comment spam.
23 comments, already:
I have been using the maven JCoverage report and it has been working great. It’s as simple as adding
maven-jcoverage-plugin to project.xml.
Nadeem (email) (link) - 17 September '04 - 21:17
Nadeem,
Is that the gpl version of JCoverage or one of the commercial versions? I evaluated the gpl version, and the documentation I saw didn’t mention that task. I also checked the jar file and didn’t see it there.
Lance Finney (link) - 17 September '04 - 23:03
I just found the JCoverage plugin page off the Maven site. I’m sorry I didn’t evaluate that, but we’re not using Maven, so it wouldn’t work for us. Thanks for the comment, though.
Lance Finney (link) - 17 September '04 - 23:08
Great article!
Nils-H - 18 September '04 - 03:56
Agreed, thanks for taking the time to review the options out there and make the information available!
idcmp - 18 September '04 - 13:25
Lance, thanks for a good writeup on EMMA. You’ve mentioned a couple of errors in the basic ANT tutorial—if you can give me details on those, I will certainly correct them.
Vlad (email) (link) - 20 September '04 - 11:30
You can introduce Hansel class by class – when you have to touch the tests for a class, instrument it with Hansel and make it run again. We are doing this at work with a product of ~500,000 LOC. It has been quite helpfull until know.
Ilja Preuß (email) - 20 September '04 - 14:19
Nice article and good comparison.
Prashant (email) - 24 September '04 - 01:33
It’s too bad you could not get JCoverage working. I think it was easy to set up. I used the provided example to modify my JUnit target. Maybe check with
http://www.javaranch.com/newsletter/2004... My only problem was that I was instrumented the unit test classes and ended up with 100% coverage right off the bat, but with some tweaking, I got it to instrument my original source and the numbers dropped. Oh well.
Jeff Conrad (email) - 08 November '04 - 12:16
Thanks for your valuable research. We were hard-pressed for time & this page helped our decision making!
Param - 02 December '04 - 02:45
I would like to know if there is a tool that could be used for J2ME-MIDP2.0 ? I tried Emma to work but Emma depends on Class Loader which J2ME-MIDP2.0 does not have.
mano (email) - 07 January '05 - 20:00
InsECT doesn’t look dead to me. The last webpage update was 2003 and the current release is 0.9.
Wonder where you looked at?
Nice report anyway.
Arndt - 25 January '05 - 06:30
You’re right, Arndt, there was a release in 2003, and they are at 0.9, not at 0.1. I don’t know if I saw an invalid page back then, or if I made a typo, or if I got InsectJ confused with something else. It is possible, though, that the sourceforge activity might have been 0% at that the point I wrote this. It’s currently 18%.
However, since InsECT hasn’t realeased in about 16 months and has such a low activity, I would still consider it inactive.
I have updated the evaluation for the 0.91 release in September 2003.
Lance Finney (link) - 25 January '05 - 06:58
This blog is a bit out of date, you should take a look at the new GroboUtils. In the new version, it’s instrumentation is literally 10x faster (estimate on my part), and the out of memory error has been fixed.
Jimmy Jarrett - 10 February '05 - 13:20
How did you get emma to run your junit task? I see there is an task but I can’t see how to get it to run junit, without reimplementing what the junit task does.
If this question is too hard to answer, please just post your build.xml that tested your unit tests with emma.
Good article.
Ricky Clarkson (email) (link) - 10 April '05 - 14:25
Ricky,
Here are the relevant snippets from our build.xml:
A necessary taskdef:
<taskdef resource=’emma_ant.properties’ classpathref=’build.classpath’/>
We modified our test target. This part goes at the top to decorate the bytecode:
<property name=”coverage.dir” value=”${basedir}/coverage”/>
<mkdir dir=”${coverage.dir}”/>
<emma enabled=”${emma.enabled}”>
<instr instrpathref=”classes.path”
destdir=”${out.instr.dir}”
metadatafile=”${coverage.dir}/metadata.emma”
merge=”true”>
<filter excludes=”
Test,
Mock”/>
</instr>
</emma>
We added these within the junit call:
<jvmarg value=”-Demma.coverage.out.file=${coverage.dir}/coverage.emma”/>
<jvmarg value=”-Demma.coverage.out.merge=true”/>
We added this at the end of the test target for cleanup:
<emma enabled=”${emma.enabled}”>
<report sourcepath=”${basedir}/src”>
<fileset dir=”${coverage.dir}”>
<include name=”*.emma”/>
</fileset>
<txt outfile=”${coverage.dir}/coverage.txt”/>
<html outfile=”${coverage.dir}/coverage.html”/>
</report>
</emma>
<delete dir=”${out.instr.dir}”/>
And here’s another target that is used to turn emma on for testing:
<target name=”emma” description=”turns on EMMA instrumentation/reporting”>
<property name=”emma.enabled” value=”true”/>
<property name=”out.instr.dir” value=”${basedir}/outinstr”/>
<mkdir dir=”${out.instr.dir}”/>
</target>
Lance Finney (link) - 12 April '05 - 10:05
Does someone has tested Rational Test RealTime ?
It seems to be a very good tool to test coverage, runtime analysis and many things else.
Any opinion ?
Nicolas (email) - 05 July '05 - 04:18
hi there,
I’ve heard about an open-source tool called Cobertura(https://sourceforge.net/projects/cobertura/,
http://cobertura.sourceforge.net/) which is in the same space as emma. Currently, it’s at v.1.4.
BR,
~A
anjan bachu (email) - 27 July '05 - 19:02
anjan,
I’ve seen good things about Cobertura. Unfortunately, I’m no longer at the same job I was at when I made this comparison, so I could not legitimately compare Cobertura to the other tools under the same conditions. I hope it’s a good tool, too.
Lance Finney (link) - 28 July '05 - 13:03
has anyone found something that works for J2ME? I have evaluated the following tools without any success: Emma, Clover, GlassJar Toolkit. I have an idea of how to actually implement one based on source code instrumentation but don’t have the time right now to do it. If anyone knows of anything else that might work please email me.
ish (email) - 12 January '06 - 23:17
Really Very Nice Analysis and this helped me a lot in decision making(which tool to choose).
Thanks a lot to you all.
Prasad Bhuvanam (email) - 29 June '06 - 03:19
I have used Cobertura (
http://cobertura.sourceforge.net/index.h..) and pretty happy with that. Good thing is its getting improved after every release and so looks promising. As of now its at version 1.8.
-Ajay KR
Ajay KR - 01 March '07 - 05:11
There are many plugins based on EMMA for all the main IDEs. For example, the
http://codecoverage.netbeans.org for Netbeans or EclEmma (and others) for Eclipse. We also did some research in this area and the conclusion was that EMMA is the best choice
Michael - 11 March '07 - 14:43
