Test Analytics With TestRoots WatchDog

Do you know how much of your development effort goes into testing? The TestRoots project’s WatchDog is an Eclipse plugin developed at TU Delft that tells you exactly how long you are working on your tests. And thanks to Eclipse Code Recommender’s latest release (2.1.11), it is now possible to install WatchDog with just 4 clicks from within your Eclipse IDE.

“How much should I test?” is a question that I often hear from my students as well as from experienced programmers. Unfortunately, I cannot give them a simple answer. So who am I? My name is Moritz Beller, and I’m pursuing a PhD at the Technical University of Delft in the Software Engineering Research Group of Prof. Arie van Deursen. Our aim in the group is to empirically explore how software engineers work in practice, and, based on this knowledge, to help them write better software.

The First Findings of WatchDog Are Alarming

In a recent study with Software Engineering students at TU Delft, we found that the students on average spend only 9% of their time testing. In fact, 50% of the students spent as little as 4% – or less! This result was especially bewildering as we enforced a strict 75% code coverage criterion on any code handed in. Clearly, when you are spending less than a tenth of your time on tests, the time in which you are willing to test and improve the quality of your product is very limited.

So, our first conclusion is that (1) code coverage does not seem to say a lot about how much effort you spent on testing.

Before the students installed WatchDog, we asked them how much time they thought they spent on testing. Maybe it was clear to them after all that they’d only spend a small fraction of their time on testing? But, again, the result was quite surprising to us: The average estimate was 27% of time for testing, 73% for code production. When you compare that with the time they actually spent (9%), you see that the students overestimated the work done on testing by a factor of three. Clearly, when you think that you are doing three times more quality assurance and testing than you actually do, this poses a big problem. A further clue to that may be that, on average, students executed only one test every hour.

Thus, our second conclusion is that (2) students’ expectation on how much they test was off by a factor of three: In reality, they test three times less than originally estimated.

What Do I Gain From Installing WatchDog?

Do you want to know whether you are falling into either trap (1) or (2)? To find out, all you have to do is install our WatchDog plugin. It supplies you with instantaneous statistics on your last hour of development in the Eclipse IDE, so that you can reconcile yourself if you overestimate the effort you put into testing. These statistics are shown in the WatchDog Statistics view.

WatchDog Statistics view

Another benefit is that you’ll receive a detailed report on your testing behaviour by mid-February 2015, which features a more in-depth evaluation than the WatchDog Statistics view. With this report you can, for example, compare your testing practices with that of other (anonymized) developers that have a similar working style as you (e.g. other open-source contributors, weekend coders, or professional developers). As a small bonus, we are raffling away awesome prizes among all the volunteers who have installed and used WatchDog. And, if you have doubts: Rest assured that WatchDog does not invade your privacy. You will stay completely anonymous, we won’t transfer any code (not even file names!), and the data we gather is purely statistical.

Introducing the Scientific Projects Extensions Marketplace

Eclipse Code Recommenders, which itself once started as a research project at TU Darmstadt, wants to support both scientists and Eclipse users eager to try out the latest technology: As of verison 2.1.11 the Code Recommenders extensions marketplace features “Scientific Projects”. WatchDog is the first Eclipse plug-in to appear under this category. In contrast to the much wider Eclipse Marketplace, the science group makes it clear that these are exclusively scientific plugins from universities and other research institutions. It therefore increases the visibility of scientific inventions to users who are interested in the latest research ideas, because scientific plugins no longer have to compete with the 11,000+ other tools in the regular marketplace.

How Do I Install WatchDog?

There are multiple ways to install WatchDog. For example, simply drag and drop our Marketplace icon into your Eclipse workspace. Alternatively, if you have Eclipse Code Recommenders 2.1.11 or later installed, visit the Code Recommenders extension marketplace:

Code Recommenders preference page

You can reach the Code Recommenders Marketplace by opening the Eclipse Preferences (Window > Preferences) and then selecting Code Recommenders. On the overview page that appears, click “Discover new extensions.”

Extensions discovery dialog

In the following window, find the “Science Projects” and select “WatchDog”.

Last but not least, you can install WatchDog via its update site, as explained in this video.

We’d be happy to see you as a WatchDogger, and hope to give you a little more insights into how you as a developer actually test your software – and how long it takes you to do that

Install from Eclipse Marketplace Install from Update Site

Add new comment