Analyze the Reliability of Your Software

ParaSoft's Jtest

Reviewed By Cynthia Paxson/ NSTL
Rating: 3 cups (good)
Version Reviewed: 1.032
Current Version: 1.033

Attention all Java™ programmers! The first automatic bug-finding system for Java™ is now available to you. The Jtest product uses ParaSoft's patent-pending Test Generation Systems Technology to analyze Java™ programs. In a nutshell, it simulates user input to analyze the reliability of the software being tested.

Because Java™ programs tend to be user driven, proper handling of erroneous input is crucial to prevent runtime exceptions. Admittedly, error checking is usually not the first priority during the development cycle of any program-- Java™ applications and applets included. Developers tend to do the fun stuff first, leaving the grunt work like error checking and testing for later. We always intend to go back and clean up after ourselves when time permits, but you all know how that story goes.

The people at ParaSoft certainly had this in mind when they engineered Jtest. This product takes a huge burden off the development staff by increasing the reliability of the Java™ program before it gets into the official testing phase. Speaking from experience, nothing is worse than working around the clock in an effort to meet unrealistic deadlines, only to find that the product is released into test before you have a chance to sufficiently test it yourself. It can be quite humiliating for the development staff and extremely frustrating for the testers. The whole experience is demoralizing.

With a product like Jtest, you can bulletproof the code as part of the development process and turn it over with confidence and pride to be tested. The testing department will love you for it. Jtest will allow them to focus on design issues and product improvements instead of on bugs being generated by invalid input, null pointers, or arithmetic and array indexing problems. Jtest locates the input that causes these kinds of problems, and points them out explicitly to the programmer so they can be fixed immediately.

Testers will no longer need to design and execute the enormous number of test scripts that are so commonly required for product verification. Instead, they can direct their efforts to more challenging issues like bench-marking and improving product-usability. Testers too often are underutilized and underestimated. They are the ones who most closely represent the end user. Their practical product knowledge is invaluable, yet they spend most of their time banging on keyboards, trying to break the product. Jtest will do a great deal of that for them. The testers can focus their attention on the areas that Jtest was unable to verify instead of on the overall product.

Let's take a look at how Jtest actually works and exactly what it can do for you. First, it is completely automated, requires no special scripting language and, does not intrude on the program being tested. To launch Jtest, one would open a DOS session and type TestApp. Jtest will start banging away on TestApp. When it discovers a bug, it reports a stack trace and a line number to the screen. You can redirect the output to a file if you prefer. My understanding is that Jtest keeps searching for paths that it can test, even after it has found an error, until it has passed through all of the iterations. After Jtest has completed the designated or default number of iterations, it writes report information to a file. The information provided by the report file is very specific. When an exception is generated, Jtest informs you of the exact line number and the type of exception. For example, the exception could be an arithmetic type like a divide by zero error, or an invalid number format type. Jtest provides the percentage of usercode tested by determining the exact number of branches in the program and how many of them have been tested. The report itself focuses on three main areas of the code: multi-condition branch coverage, method coverage, and constructor coverage. For each case, the report defines how many instances exist, and of these, how many were tested. The report contains an annotated source file that denotes which lines generated an exception and why, which lines lines were partially tested, and which lines were fully tested.

Playback capabilities are provided allowing you to automatically execute the sequence of events that caused the exception. Gone are the days of being unable to reproduce a specific bug sequence! If there were 10 failures, there will be 10 different iterations available for playback. The player is invoked by typing player TestApp x, with x being the iteration of interest. A window opens with very readable stack information. You can then single step through the sequence of events or just play them back. Pause is also available in case you wish to examine the screen at an exact point. The program will be instantiated, and the simulated keystrokes will be played back. You can watch exactly how the program operates, as if a real user were sitting at the keyboard. You will see the exception occur and witness how your program handles it.

If you are testing an applet, Jtest produces an .html file for the exceptions that it finds. This allows you to examine and modify the .html for your specific testing needs. You can then invoke the .html file via Sun's AppletViewer utility to see how the applet responds without Jtest running. Experience has taught every developer I know not to completely rely on test tools. Some have been known to cause more problems than they find. For those of us who are skeptical of outside test software, being able to remove Jtest from the loop offers proof that the problem was not caused by Jtest The .html files generated by Jtest can be saved and used to verify that the bug has been fixed without running Jtest through all iterations again.

Jtest is most definitely a technical product designed for techies by techies. It currently does not wrap itself in a fashionable graphic interface. It does not automatically install. It requires the user to work in a DOS session. It is limited to programs that are between 3k and 5k. Its report generator is crude but sufficient. The folks at ParaSoft are definitely aware of these issues and are currently addressing them. My hunch is that ParaSoft was more interested in bringing this technology to the market than wrapping it up in a fancy interface. The only real drawback to the product is its size limitations.

Most technical people are not afraid of a little DOS session or hand-carving environment variables. Personally, I prefer to do these things myself so that I know exactly what is going on in my system. Nothing annoys me more than installing a piece of software that takes over my universe and treats itself as king. Jtest respects its users. If you can make use of this technology, this product is for you.

By the way, I ran Jtest against a couple of different sample programs provided by Sun with their JDK. Jtest found at least one exception in each case I tried, so don't be insulted if it brings your software to its knees the first couple of times through. That's its purpose!

This review appeared in the April 1998 edition of Java™ Report.