Lesson 3 - Performing Black-Box Testing

Black-box (functionality) testing checks a class's functionality by determining whether or not the class's public interface performs according to specification. This type of testing is performed without paying attention to implementation details.

If your class contains Design by Contract (DbC)-format specification information, Jtest completely automates the black-box testing process. If not, Jtest makes the black-box testing process significantly easier and more effective than it would be if you were creating test cases by hand.

Jtest reads specification information built into the class with the DbC language, then automatically develops test cases based on this specification. Jtest designs its black-box test cases as follows:

• If the code has postconditions, Jtest creates test cases that verify whether the code satisfies those conditions.
• If the code has assertions, Jtest creates test cases that try to make the assertions fail.
• If the code has invariant conditions (conditions that apply to all of a class's methods), Jtest creates test cases that try to make the invariant conditions fail.
• If the code has preconditions, Jtest tries to find inputs that force all of the paths in the preconditions.
• If the method under test calls other methods that have specified preconditions, Jtest determines if the method under test can pass non-permissible values to the other methods.

Jtest also helps you create black-box test cases if you do not use Design by Contract. You can use Jtest's automatically-generated set of test cases as the foundation for your black-box test suite, then extend it by adding your own test cases.

Test cases can be added in a variety of ways; for example, test cases can be introduced by adding:

• Method inputs directly to a tree node representing each method argument.
• Constants and methods to global or local repositories, then adding them to any method argument.
• JUnit-format Test Classes for test cases that are too complex or difficult to be added as method inputs.

If a class references external resources, you can enter your own stubs or have Jtest call the actual external method.

When the test is run, Jtest uses any available stubs, automatically executes the inputs, and displays the outcomes for those inputs in a simple tree representation. You can then view the outcomes and verify them with the click of a button. Jtest automatically notifies you when specification and regression testing errors occur on subsequent tests of this class.

This tutorial covers the following main topics:

• Adding the Path to Your JDK
• Performing Automatic Black-Box Testing With Design by Contract
• Using Test Classes
• Using Test Classes to Test Methods That Accept Object Type Inputs
• Using Test Classes to Test a Specific Calling Sequence
• Using Test Classes to Validate the State of an Object
• Using JUnit Test Classes
• Adding Primitive Inputs to Static and Instance Methods

Adding the Path to Your JDK

To perform black-box testing with Jtest, you must have the JDK compiler on your PATH. The 'javac' command is the compiler; it is usually located under the bin directory of most JDK installations. To tell Jtest where your Java compiler is located:

1. Click the Global button in the current Jtest UI.
2. In the Global Test Parameters window, double-click the Common Parameters> Path to JDK directory, then enter the path to your JDK in the dialog box that opens.
   
4. Click OK.

Performing Automatic Black-Box Testing With Design by Contract

This example demonstrates how Jtest performs automatic black-box testing on Example.class, a class file that contains Design by Contract-format specification information.

To test this class, perform the following steps:

1. Go to Jtest's Class Testing UI. (This UI opens by default when you launch Jtest).
2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test.
3. Browse to Example.class (in <jtest_install_dir>/examples/eval) using the Browse button in the Class Name panel.
4. Click the Start button in the tool bar.

Jtest will perform both static and dynamic on the class. Because the code's specification is incorporated into the code using Design by Contract comment tags, Jtest can fully automate black-box (functionality) testing as well as white-box (construction) testing. Jtest will automatically create and execute black-box test cases that verify the functionality described in the class's postcondition contract. It will also create and execute white-box test cases that check how the class handles a wide range of inputs.

A dialog box will open to notify you when testing is complete. Information on test progress will be displayed in the Test Progress panel. This test uncovers one static analysis error, one Design by Contract violation and one uncaught runtime exception.

The following Design by Contact violation will be reported in the Design by Contract Violations branch of the Errors Found panel.

This violation indicates that one of the class's methods did not function as described in the specification. To see more information about this violation, expand the violation's branch.

To open the source code of the class in an editor, right-click the Source button, then choose Edit Source from the shortcut menu.

The source file reveals that the code's @post contract (postcondition) requires the method to return the value of a+b. However, as Jtest revealed, it does not function as specified: the method actually returns the value of a-b. If this were your own class, you would now fix the problem, recompile the class, then retest it to check if your modifications fixed the problem.

To see a sample of the test cases that Jtest automatically created to test this class's functionality, click the View button to open the View Test Cases window. In the View Test Cases window, Control-right-click the Automatic Test Cases node, then choose Expand Children from the shortcut menu.

Next, go to the uncaught runtime exception found (located in the Uncaught Runtime Exceptions branch of the Errors Found panel) and expand its branches.

This error message shows that a NegativeArraySizeException occurs when a negative index is used as an index to an array. This is an expected exception. If this were your code, you would want to document this exception in your code by adding the following Design by Contract Javadoc comment above the method:

 /** @exception java.lang.NegativeArraySizeException */

By adding this comment, you make the code easier to maintain. Someone looking at the code later on will immediately know that the method is throwing an exception because the code is supposed to throw an exception, not because the code has a bug. In addition, you tell Jtest to suppress future occurrences of this exception.

Using Test Classes

Test Classes let you test cases that are too complex or difficult to be added as method inputs. For example, you might want to use Test Classes if you want to:

• Use objects as inputs for static and instance methods.
• Test a calling sequence and check the state of the object using asserts.
• Create complicated test cases that depend upon a specific calling sequence.
• Validate the state of an object.

A Test Class is a class that extends jtest.TestClass and is used to specify test cases that Jtest should use to test the class. For information about jtest.TestClass, see the Jtest API javadoc (you can access this documentation by choosing Help> Jtest API). The jtest.TestClass file is in the jtest.zip file located in <jtest_install_dir>/classes.

Test Classes give you an easy way to write complicated test cases and to verify the state of the objects under test. You can write your own Test Classes using any Java development environment, or you can integrate JUnit classes into your Jtest tests.

This example demonstrates how to add and execute the test cases contained in the CounterTest Test Class located in <jtest_install_dir>examples/dynamic/testclasses to test the Counter class contained in that same directory.

Before you start performing tests, open the CounterTest's source code in your preferred source editor.

Note the following things about this class:

• It extends jtest.TestClass in <jtest_install_dir>/classes/jtest.zip.
• It has two test cases, method testAdd, method testSubstract, and a test case method called `testDivideThrowsArithmeticException.
• Each of its test case methods begins with the word `test'.
• It uses asserts to validate that the calling sequence was performed.

To test the Counter class with the CounterTest Test Class, perform the following steps:

1. In the Class Testing UI, browse to and open <jtest_install_dir>/examples/dynamic/testclasses/Counter.class.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined. You will see that Jtest has automatically detected the examples.dynamic.testclasses.CounterTest Test Class. (Jtest automatically detects Test Classes if they are named "<name_of class_under_test>Test" (for example, "fooTest") and if they are located in the same package as the class under test.
   
   
4. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
5. When the test is complete, view the results of the user defined test cases.
   1. Click View to open the View Test Cases window.
   2. Open the Test Case branches located under User Defined Test Cases> Test Classes. This shows you that Jtest executed the test cases contained in the Test Class.

In the Errors Found panel, notice that Jtest finds one Uncaught Runtime Exception from the automatic test cases and one specification error from the testSubstract user-defined test case in CounterTest.

Using Test Classes to Test Methods That Accept Object Type Inputs

The following examples use Test Classes to specify test cases for static methods and instance methods that accept object type arguments.

Testing a Static Method that Accepts an Object Type Input

This example demonstrates how to use a Test Class for a static method that accepts a String argument. This example uses the following Test Class (located in <tutorial_install_dir>/lesson3/StaticExampleTest.class).

 import jtest.*;
 
 public class StaticExampleTest extends TestClass {
 
 public static void testReturnStaticString() {
 	 assert( "String should be hello",
 	 StaticInstanceExample.returnStaticString( "hello" ) 
 	 == "hello");
 }
 }
 

Note the following things about this class:

• It extends jtest.TestClass (located in <jtest_install_dir>/classes/jtest.zip.
• It contains one test case (method testReturnStaticString).
• It uses an assert to validate whether the correct object is returned. (To learn more about the assert method, choose Help> Jtest API).

To test the StaticInstanceExample class with the StaticExampleTest Test Class, perform the following steps.

1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/StaticInstanceExample.class.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined.
4. Right-click Test Classes, then choose Add Test Class from the shortcut menu.
5. Type StaticExampleTest in the text field, then press Enter.
   
   
   
6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
8. When the test is complete, view the results of the user defined test case.
   1. Click View to open the View Test Cases window.
   2. Open User Defined Test Cases> Test Classes> Test Case 1.

Testing an Instance Method that Accepts an Object Type Input

This example demonstrates how to use a Test Class for an instance method that accepts a String argument. The example Test Class used for this example is located in <tutorial_install_dir>/lesson3/InstanceExampleTest.class).

To run this test, perform the following steps:

1. If you have not already done so, browse to and open <tutorial_install_dir>/lesson3/StaticInstanceExample.class.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined.
4. Right-click Test Classes, then choose Add Test Class from the shortcut menu.
5. Type InstanceExampleTest in the text field, then press Enter.
   
6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
8. When the test is complete, view the results of the user defined test case.
   1. Click View to open the View Test Cases window.
   2. Open User Defined Test Cases> Test Classes> Test Case 1.

Using Test Classes to Test a Specific Calling Sequence

Test Classes can be used to test a calling sequence and check the state of the object using asserts. They can also be used to create complicated test cases that depend upon a specific calling sequence.

This example demonstrates how to use Test Classes to test a calling sequence. This example uses the following Test Class (located in <tutorial_install_dir>/lesson3/CallingSequenceTest.class).

 
 import jtest.*;
 
 public class CallingSequenceTest extends TestClass{
 
 public static void testSequence() {
 	 CallingSequence m = new CallingSequence( 0 ); 
 	 // ensure the total after the following sequence is '20' 
 	 m.add (10); 
 	 m.mult (2); 
 	 m.add (2); 
 	 m.sub (2); 
 	 assert ("total should be 20", m.getTotal () == 20); 
 } 
 }     
 

To test the CallingSequence class with the CallingSequenceTest Test Class, perform the following steps:

1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/CallingSequence.class.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined.
4. Right-click Test Classes, then choose Add Test Class from the shortcut menu.
5. Type CallingSequenceTest in the text field, then press Enter.
   
   
6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
8. When the test is complete, view the results of the user defined test case.
   1. Click View to open the View Test Cases window.
   2. Open User Defined Test Cases> Test Classes> Test Case 1.

Using Test Classes to Validate the State of an Object

This example demonstrates how to use a Test Class to validate the state of an object in the java.util.Vector class from the JDK distribution's rt.jar file. The example Test Class is located in <tutorial_install_dir>\lesson3\MyVectorTest.java):

 
 import java.util.Vector;
 
 public class ThisVectorTest extends jtest.TestClass {
 
   public static void test1 () {
     Vector vector = new Vector ();
     vector.addElement ("name");
     assert ("should be 1", vector.size () == 1);
   }
 
   public static void test2 () {
     Vector vector = new Vector ();
     vector.addElement ("name");
     vector.addElement ("address");
     vector.addElement ("phone number");
     assert ("size should be 3", vector.size () == 3); 
   }
 /*
   public static void testSetSizeThrowsExceptionOnNegative () {
     Vector vector = new Vector ();
     try {
       vector.setSize (-1);
     } catch (ArrayIndexOutOfBoundsException e) {
 	 return;
     }
     // should not reach this point in the test case
     assert ("should have thrown exception: size is negative", 
false);
   }
 */
 
 }

Note the following things about this class:

• It extends jtest.TestClass in <jtest_install_dir>/classes/jtest.zip.
• It has two test cases, method `test1' and method `test2', and a commented test case method called 'testSetSizeThrowsExceptionOnNegative'.
• It uses asserts to validate the state of the vector after operations have been performed on it.
• Each test case begins with `test'.

To test the class using this Test Class. perform the following steps:

1. In the Class Testing UI, click New, then enter
   java.util.Vector
in the Class Name field.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Common Parameters> java/javac-like parameters> -cp.
4. Right-click -cp, then choose Edit from the shortcut menu.
5. Browse to and select <tutorial_install_dir>/lesson3/, then click Add.
   
6. In the Class Test Parameters window, open Dynamic Analysis> Test Case Execution.
7. Clear the Execute Automatic option.
8. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined.
9. Right-click Test Classes, then choose Add Test Class from the shortcut menu.
10. Type
ThisVectorTest
in the text field, then press Enter.
    
    
11. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
12. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
    • If you see a dialog box that asks you to set a path to Vector.java, click Cancel. The source code for java.util.Vector is not required for this exercise.
13. When the test is complete, two errors should be reported.
    
    To view the results of the user defined test cases:
    1. Click View to open the View Test Cases window.
    2. Open Test Case branches found under User Defined Test Cases> Test Classes.
       If you look in each test case's Outcomes node, you will see that no errors were found for these test cases.

To add a new test case that will throw an exception as an outcome, perform the following steps:

1. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Test Classes.
2. Right-click ThisVectorTest, then choose Edit Test Class from the shortcut menu.
   
   
3. In the text editor that contains the Test Class, uncomment the 'testSetSizeThrowsExceptionOnNegative' method, then save and close the Test Class file.
4. In the Class Test Parameters window, right-click ThisVectorTest, then choose Compile Test Class from the shortcut menu.
5. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
6. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.

To see what happens when an exception fails, perform the following steps:

1. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Test Classes.
2. Right-click ThisVectorTest, then choose Edit Test Class from the shortcut menu.
   
   In the text editor that contains the Test Class, comment out the return statement of the catch block, then save and close the Test Class file.
3. Right-click ThisVectorTest, then choose Compile Test Class from the shortcut menu.
4. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
5. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. An exception will appear in the Specification and Regression Errors branch of the Errors Found panel.

Using JUnit Test Classes

Jtest can use JUnit Test Classes as well as Jtest Test Classes. For a demonstration of how to integrate your JUnit Test Classes into Jtest, see Lesson 7- Using JUnit Test Classes with Jtest.

Adding Primitive Inputs to Static and Instance Methods

This section contains examples that demonstrate how to add primitive inputs to static and instance methods.

Adding Primitive Inputs to a Static Method

This example demonstrates how to add primitive input values to static methods and how to validate the outcomes for these inputs.

1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/StaticInstanceExample.class.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Method Inputs> returnStaticInt.
4. Right-click int ARG1, then choose Add Input Value from the shortcut menu.
5. Type 3 in the text field, then press Enter.
   
6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
8. When the test is complete, view the results of the user defined test case, then validate the outcome.
   1. Click View to open the View Test Cases window.
   2. Open User Defined Test Cases> Method Inputs> returnStaticInt> Test Case 1.
   3. Right-click RETVAL=3, then choose Mark as Correct to indicate that this is the correct outcome for the given input.

Adding Primitive Inputs to Instance Methods

This example demonstrates how to add primitive inputs to instance methods and how to validate the outcomes for these inputs.

1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/StaticInstanceExample.class.
2. Click the Class button to open the Class Test Parameters window.
3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Method Inputs> returnInstanceInt.
4. Right-click StaticInstance Example THIS, then choose Add Input Value from the shortcut menu.
   
   
5. Type new StaticInstanceExample () in the text field, then press Enter.
6. Right-click int ARG1, then choose Add Input Value from the shortcut menu.
7. Type 5 in the text field, then press Enter.
8. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu.
   
9. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI.
10. When the test is complete, view the results of the user defined test case, then validate the outcome.
    1. Click View to open the View Test Cases window.
    2. Open User Defined Test Cases> Method Inputs> retunInstanceInt> Test Case 1.
    3. Right-click RETVAL=5, then choose Mark as Correct to indicate that this is the correct outcome for the given input.