Search

ParaSoft

HOME
PRODUCTS
SUPPORT
ABOUT
WHAT'S NEW
EVENTS


Insure++

Quick facts

Add-on Modules:
   -INUSE
   -TCA

Comparisons

Technical Papers

Support & Manuals

FAQs

Recent Reviews

User Testimonials

Press Releases


Insure tool to debug c++

TCA - Using TCA


Table of Contents


Using TCA

Preparing your code for coverage analysis

The coverage analysis system works in a similar way to Insure++. Your application is processed with the insure command instead of your normal compiler. This process first builds a temporary file containing your original source code modified to include checks for coverage analysis, and then passes it to your normal compiler. While your application runs, these modifications cause your application to create a database containing information about which blocks were executed. This database can then be analyzed with a special application (tca) to create reports.

Since coverage analysis is such a powerful tool when used in conjunction with Insure++, it is automatically enabled whenever you compile an application with the insure command, unless you specifically disable it, as described in the Configuration Files section of the of this manual. If you only want to do coverage analysis (i.e. you don't want the normal Insure++ runtime checking), you can compile and link with the -Zcov switch.

Warning

Please note that if you compile with insure and the -Zcov option, you must also link with the -Zcov option, and vice versa. You cannot use -Zcov in only one stage of the build.

An example - sorting strings

To see this process in action, consider the code shown in Figure 1, which is a modified version of the bubble sorting algorithm used in previous examples.


    1:	/* 
    2:	 * File: strsort.c
    3:	 */ 
    4:	#include <stdio.h>
    5:	#include <string.h>
    6:	
    7:	bubble_sort(a, n, dir)
    8:		char **a;
    9:		int n, dir;
    10:	{
    11:		int i, j;
    12:	
    13:		for(i=0; i<n; i++) {
    14:			for(j=1; j<n-i; j++) {
    15:		 		if(dir * strcmp(a[j-1], a[j]) > 0) {
    16:					char *temp;
    17:			
    18:					temp = a[j-1];
    19:					a[j-1] = a[j];
    20:					a[j] = temp;
    21:				}
    22:			}
    23:		}
    24:	}
    25:	
    26:	main(argc, argv)
    27:		int argc;
    28:		char **argv;
    29:	{
    30:		int i, dir, length, start;
    31:	
    32:		if (argc > 1 && argv[1][0] == '-') {
    33:			if (argv[1][1] == 'a') {
    34:				dir = 1; length = argc-2; start = 2;
    35:			} else if (argv[1][1] == 'd') {
    36:		 		dir = -1; length = argc-2; start = 2;
    37:			}
    38:		} else {
    39:			dir = 1; length = argc; start = 1;
    40:		}
    41:		bubble_sort(argv+start, length, dir);
    42:		for (i = 2; i < argc; i++) 
    43:			printf("%s ", argv[i]);
    44:		printf("\n");
    45:		return 0;
    46:	}

Figure 1. Sorting example source code (strsort.c)

The basic idea of this program is to sort a set of strings supplied as command line arguments in either ascending or descending order, according to the settings of the command line switches.

strsort -a s1 s2 s3 ...
Sorts strings in ascending order.
strsort -d s1 s2 s3 ...
Sorts strings in descending order.
strsort s1 s2 s3 ...
Sorts strings in ascending order.

If you wish to try the commands shown in this section, you can use the source code supplied with the Insure++ examples by executing the command

    cp /usr/local/insure/examples/c/strsort.c . 

To compile and execute this program with both runtime error detection and coverage analysis enabled, simply use the normal insure command

    insure -g -o strsort strsort.c 

In addition to compiling the program, this will create a file called tca.map which describes the layout of your source file. We can now perform a set of tests on the application. A few samples are shown in Figure 2. The statements beginning with the $ symbols are the commands executed and the remaining text is the response of the system.


    $ strsort -a aaaa bbbb
    aaaa bbbb
    ** TCA log data will be merged with tca.log **

    $ strsort -a bbb aaa
    aaa bbb
    ** TCA log data will be merged with tca.log **

    $strsort -d aaa bbbb
    bbbb aaa
    ** TCA log data will be merged with tca.log **

    $ strsort -d bbb aaa
    bbb aaa
    ** TCA log data will be merged with tca.log **

Figure 2. Sample test runs of the string sorting program

Note the following features from the figure.

Analyzing the test coverage data

Analysis of the test coverage data is performed using the tca command. There are several summary levels. The ones we will introduce here are:

Overall summary (default)
Shows the percentage coverage at the application level - i.e., summed over all program entities.

Function summary (-df switch)
Displays the coverage of each individual function in the application.

Source code summary (-ds switch)
Displays the source code with each block marked as executed (.) or not (!).

To see these commands in action, execute the command

    tca tca.log

This displays the top level summary, shown below.

     COVERAGE SUMMARY
     ================
        1   block untested
       12   blocks tested

92% covered

As can be seen, the overall coverage is quite high. However, one program block remains untested. To find out which one is untested, execute the command

    tca -df tca.log

This command displays the function level summary, and includes functions that are 100% tested.

     COVERAGE SUMMARY - by function
     ================

blocks blocks %cov = functions sorted untested tested %tested by name ------------------------------------------------------ 0 4 100% bubble_sort [strsort.c, line 7-13] 1 8 88% main [strsort.c, line 26-45]

From this listing, we can see that the function bubble_sort is completely tested, but that one block in main remains untested. To find out which one, we execute the command

    tca -ds tca.log

This results in the following output.

     UNTESTED BLOCKS - by file
     ===============
     
FILE strsort.c 92% covered: 1 untested / 12 tested
/* * File: strsort.c */ ... main(argc, argv) int argc; char **argv; { int i, dir, length, start;
. -> if (argc > 1 && argv[1][0] == `-`){ . -> if (argv[1][1] == `a`){ . -> dir = 1; length = argc-2; start = 2; . -> } else if (argv[1][1] == `d`){ . -> dir = -1; length = argc-2; start = 2; } } else { ! -> dir = 1; length = argc; start = 1; } . -> bubble_sort(argv+start, length, dir); for (i = 2; i < argc; i++) . -> printf("%s ", argv[i]); . -> printf("\n"); return 0; } ...

This listing shows exactly which statements have been executed and which have not. We can see from the results that the untested block corresponds to the case where strsort is executed with neither the -a nor -d command line switches.

Achieving 100% test coverage

The previous analysis tells us that we can achieve 100% test coverage in this example by executing the strsort program without either of its two switches.

To complete testing, therefore, we can execute the program with the command

    strsort aaa bbb

This produces the following output

     [strsort.c:15] **READ_NULL**
     >> if(dir * strcmp(a[j-1], a[j]) > 0) {

Reading null pointer: a[j]
Stack trace where the error occurred: bubble_sort() strsort.c, 15 main() strsort.c, 41
**Memory corrupted. Program may crash!!**
** TCA log data will be merged with tca.log **

which indicates that Insure++ has found an error in this code block.(Footnote 1)

However, if we now execute the command tca tca.log, we see the following output

	 COVERAGE SUMMARY
         ================
              0   blocks untested
             13   blocks tested

100% covered

which indicates that the application has now been 100% tested.

This means that the set of test cases that we have now run, including this last one, completely exercises all the code in this application. It would now make sense to incorporate these test cases (in conjunction with Insure++) into a quality assurance test suite for this code.

There are several .psrc options you can use to control coverage analysis. These are documented in the Configuration Files section.

How to use coverage analysis

If you are using Insure++, coverage analysis information will be automatically built into your program. At any time after you have run your code you can use the tca command to find any blocks which have not been executed. For clarity, the process is broken down into three steps below, along with important information about what's happening at each stage.

  • Compiling applications and building their coverage analysis database (usually named "tca.map").
  • Running test cases against applications that have been compiled with coverage analysis enabled, which creates entries in the TCA log file (usually named "tca.log").
  • Running the TCA analysis tool to generate coverage analysis reports from the given coverage log file(s).

You can make your program displays a coverage analysis report when it exits by adding the

    insure++.summarize coverage

option to your .psrc file. The coverage_switches option lets you set flags to control the output just as though you were passing those switches to tca.

Step 1: compile time

At compile time, Insure++ creates a database of each file processed, describing how many blocks and statements are in each file and function. This database is called a map file, because it provides TCA with a map of how your program is laid out. By default, the name of this file is tca.map, but you can change the name of this file by adding a coverage_map_file option to your .psrc file.

Ideally, all the files in your application should store their information in the same map file. If your source code is spread across several directories, you will probably want to set the map filename using a full path, e.g.

    insure++.coverage_map_file ~/project.map

If you compile several files simultaneously and they are all trying to modify the same map file, you may end up with a corrupt map file. In this case, you will need to delete the original map file and recompile the application you are interested in.

As mentioned earlier, if you are only interested in coverage information and not debugging, you can add the -Zcov option to the insure command lines that build your program. Remember to use -Zcov consistently, i.e. at both compilation and linking, if you use it at all.

Step 2: runtime

At runtime, your program (compiled with Insure++) writes a log file, which records the blocks that were actually executed during a specific run. By default, this file is called tca.log, but as with the map file, you can change the name of this file by adding a coverage_log_file option to your .psrc file. Normally, each time you run your program the new log information will be combined with any found in the existing log file, unless the data is not compatible (because you changed your code and recompiled, for example).

Another useful option is to generate a new log file each time your application runs. You can do this by taking advantage of the %n filename option, for example

    insure++.coverage_log_file tca.log.%n

In this example, each run would make a new file, such as tca.log.0, tca.log.1, and so forth. If your program forks, you will need to use this option so that each child creates its own log file.

Step 3: using TCA to display information

After you have created one or more log files, you can use the tca command to get the information in which you are interested. tca normally sends its reports to stdout. If you would like to use the graphical version to generate coverage reports, please see page 23. You can specify any number of log files on the command line, and TCA will combine the data before displaying the results. (If the log files are not compatible, e.g. because they are from different applications, TCA will throw out the ones which don't match the first log file).

TCA will also need to read the map file created at compilation time. Since this name is stored in the log file, you won't normally need to specify it.

By default, TCA will give you a quick summary of how much of your code was tested. Using different options, you can get detailed reports of coverage by file, function, or even block. For each block, TCA can tell you how many times it was executed, summing over all the log files (unless coverage_boolean was on at compile time, the default setting).

Note that if a single statement spans several lines of source code, TCA treats the statement as lying on the last line - this is only important for understanding the output of TCA, and does not effect how coverage statistics are calculated.

How are blocks calculated?

Unlike some other coverage analysis tools which work on a line-by-line basis, TCA is able to group your code into logical blocks. A block is a group of statements which must always be executed as a group. For example, the following code has three statements, but only one block.

    i = 3;
    j = i+3;
    return i+j;

Some of the advantages of using blocks over lines are

  • lines of code which have several blocks are treated separately
  • grouping equivalent statements into a single block reduces the amount of data you need to analyze
  • By treating labels as a separate group, you can actually detect which paths have been executed in addition to which statements.

Note that conditional expressions containing && or || are grouped with the statement they are part of. Also, the three elements of a for loop are treated as part of the for statement (e.g. e1, e2, and e3 in the code fragment for(e1;e2;e3)).

Figures 3, 4, and 5 illustrate the above concepts with a simple test program.

Figure 3 shows the original code and how the blocks are determined. In this particular example, there are 16 blocks.

TCA Screen capture
Figure 3. Sample program divided into blocks

Figure 4 shows the output of "tca -ct -ds tca.log" after several test runs with different values (test ; test 2 ; test 2 ; test 4 ; test 7 ; test 3). By looking at this, you can see which paths have been executed and which have not. Notice that counts are only given at the beginning of each block, and not for each statement within each block.


BLOCK USAGE - by file
============

FILE test.c 87% covered: 2 untested / 14 tested


		#include <ctype.h>

		main(int argc, char **argv) {
			int flag;
	6 ->		if (argc &l;t 2 || !isdigit(argv[1][0])) {
	1 ->			printf(\`Bad argument(s)\n\');
				exit(1);
			}
	5 ->		switch(atoi(argv[1])) {
	0,2,1 ->		case 1: case 2: case 3:
	3 ->			flag = 1;
				break;
		1 ->	case 4:
	0 ->		case 5:
	1 ->			flag = 2;
				break;
	1 ->		default:
	1 ->			flag = 0;
				break;
			}
	5,4,1 ->	if (flag > 0) flag = 1; else flag = 0;
	5 ->		printf(\`Flag is %s\n\', flag ? \`1\' : \`0\');
			exit(0);
		}

Figure 4. Output from "tca -ct -ds tca.log"

Finally, Figure 5 shows the terser output of "tca -ds tca.log". In this instance, only blocks which have not been executed are marked (corresponding to blocks with a count of zero in the previous figure). The "!" character symbolizes not executed. For lines with multiple blocks, you will also see the "." character which means that group was executed. This is so you can easily identify which blocks on that line were not tested. Once again, only the first line of code within each block will be marked in this fashion. Of you are using the graphical TCA, the first line of the block will be colored as executed (red) while the rest of the block will be colored as not executed (black).


    UNTESTED BLOCKS - by file
    ===============

    FILE test.c 87% covered: 2 untested / 14 tested


              #include <ctype.h>

              main(int argc, char **argv) {
                 int flag;

         . ->  if (argc < 2 || !isdigit(argv[1][0])) {
         . ->     printf(\`Bad argument(s)\n\');
                    exit(1);
                 }
         . ->  switch(atoi(argv[1])) {
       !.. ->  case 1: case 2: case 3:
         . ->     flag = 1;
                    break;
         . ->  case 4:
         ! ->  case 5:
         . ->     flag = 2;
                    break;
         . ->  default:
         . ->     flag = 0;
                    break;
                 }
       ... ->  if (flag > 0) flag = 1; else flag = 0;
         . ->  printf(\`Flag is %s\n\', flag ? \`1\' : \`0\');
                 exit(0);
              }

Figure 5. Output from "tca -ds tca.log"

The TCA Display

The TCA display is a graphical representation of the reports generated during run time. By utilizing this tool, you will be able to view your tca.log files with ease. Much like Insra, TCA allows you to load and save files, browse through source code, and even access on-line help.

TCA
Figure 6. Sample TCA display

The TCA display may be invoked by calling tca with the -X switch along with any other command line options. The following subset of TCA command line options are meaningful for the graphical tool, while the remaining unsupported ones are silently ignored.

Command line options:

Command
Explanation
-df
Display by function
-do
Display by object/class
-dF
Display by file
-dd
Display by directory
-ns
Simple function names
-ne
Extended function name - include argument types
-nm
Include modifiers const/volatile in function arguments
-ff name
Only show coverage related to function "name". This option only applies when -df or -dF is specified.
-fo name
Only show coverage related to object "name". This option only applies when -do is specified.
-fF name
Only show coverage related to file "name". "name&qout must include the full path to the file. This option only applies when -dF -df is specified.
-fd name
Only show coverage related to directory "name". This option only applies when -dd is specified.
-s {keys}
Sort output by keys (d,F,n,%,#,b,1)
-ct
Show hit counts in the source browser

Loading a report file

By default, TCA displays a report based on the log files that were included in the command line when the program was started. Coverage statistics from additional files may be included in the report by clicking on the Load button and selecting a new log file. The data contained in the newly selected log file will be combined with the existing data and a new report will be generated.

Browsing the source

The Browse button generates a new windows containing the next level of coverage detail. For example, if you are currently displaying a report "by directory", clicking Browse will open a new window displaying a report "by file" for that directory. If you were to click Browse again, you would get another new window showing a report "by function" for the file(s). Clicking Browse a final time will display the actual source code itself, annotated with coverage information for each block.

Double-clicking on a line in the display is the equivalent to selecting a line and clicking the Browse button.

Inuse

Figure 7. Browsing Annotated Source Code.

Reports

The level of detail displayed may be changed by clicking on the Reports button. A dialog box will appear, allowing you to choose from one of four report types: by directory, by file, by function, or by class.

Sorting

The order in which the coverage information is presented may be modified by clicking on the Sort button. A dialog box will appear in which you can enter the sort keys to be used. Any combination of the following keys may be used:

Sort Keys
Explanation
d
by directory
F
by file name
f
by function
%
by percent covered
#
by number of hits
b
by number of blocks

For example, in order to sort by percent covered and decide collisions by the function name, the sort key string should be "%f".

The current sort key string is displayed on the status bar.

Message

This button becomes active when TCA cannot perform a given task. Pressing it will bring up a window that describes the error(s).

Help

Clicking on the Help button invokes context-sensitive help for TCA. The shape of the cursor will change to a question mark/arrow combination. Clicking on a region of the display will then cause the Help window to appear with a description of the relevant features.

Building a Test Suite

Now that you have all this coverage analysis information, what's the best way to use it? Typically, you will have several tests for your code designed to exercise various features. Together, these tests make a test suite. After you have run your tests, use the tca command to discover which blocks have not been executed. This will indicate deficiencies in your test suite.

At this point, you should create more tests to try and exercise the code that was missed by your test suite so far. After you have created more tests, you can repeat the process. If the goal of 100% coverage is unreachable, you will need to make a subjective decision about how thorough you can afford to be.

Inuse

Figure 8. An iterative approach to building a test suite.


Footnotes

(1)
Finding and fixing this error is left as an exercise for the reader. The reader should also remember that if she built the program with the -Zcov option, this bug would not have been detected by Insure++.
Tools to debug c++ and java
(888) 305-0041 info@parasoft.com Copyright © 1996-2001 ParaSoft