TCA - Using TCA
Table of Contents
The coverage analysis system works in a similar way to
Insure++.
Your application is processed with the insure
command instead of your normal compiler. This process first builds a
temporary file containing your original source code modified to include
checks for coverage analysis, and then passes it to your normal compiler.
While your application runs, these modifications cause your application
to create a database containing information about which blocks were
executed. This database can then be analyzed with a special
application (tca) to create reports.
Since coverage analysis is such a powerful tool when used in conjunction
with Insure++, it is automatically enabled whenever you compile
an application with the insure command, unless
you specifically disable it, as described in the
Configuration Files section of the
of this manual. If you only want to do
coverage analysis (i.e. you don't want the normal Insure++
runtime checking), you can compile and link with the
-Zcov switch.
Please note that if you compile with insure
and the -Zcov option, you must also link
with the -Zcov option, and vice versa. You
cannot use -Zcov in only one stage of the
build.
To see this process in action, consider the code shown in
Figure 1, which is a modified version of
the bubble sorting algorithm used in previous
examples.
1: /*
2: * File: strsort.c
3: */
4: #include <stdio.h>
5: #include <string.h>
6:
7: bubble_sort(a, n, dir)
8: char **a;
9: int n, dir;
10: {
11: int i, j;
12:
13: for(i=0; i<n; i++) {
14: for(j=1; j<n-i; j++) {
15: if(dir * strcmp(a[j-1], a[j]) > 0) {
16: char *temp;
17:
18: temp = a[j-1];
19: a[j-1] = a[j];
20: a[j] = temp;
21: }
22: }
23: }
24: }
25:
26: main(argc, argv)
27: int argc;
28: char **argv;
29: {
30: int i, dir, length, start;
31:
32: if (argc > 1 && argv[1][0] == '-') {
33: if (argv[1][1] == 'a') {
34: dir = 1; length = argc-2; start = 2;
35: } else if (argv[1][1] == 'd') {
36: dir = -1; length = argc-2; start = 2;
37: }
38: } else {
39: dir = 1; length = argc; start = 1;
40: }
41: bubble_sort(argv+start, length, dir);
42: for (i = 2; i < argc; i++)
43: printf("%s ", argv[i]);
44: printf("\n");
45: return 0;
46: }
Figure 1. Sorting example source code (strsort.c)
The basic idea of this program is to sort a set of strings supplied
as command line arguments in either ascending or descending order,
according to the settings of the command line switches.
strsort -a s1 s2 s3 ...
- Sorts strings in ascending order.
strsort -d s1 s2 s3 ...
- Sorts strings in descending order.
strsort s1 s2 s3 ...
- Sorts strings in ascending order.
If you wish to try the commands shown in this section, you can use
the source code supplied with the Insure++ examples by executing
the command
cp /usr/local/insure/examples/c/strsort.c .
To compile and execute this program with both runtime error detection
and coverage analysis enabled, simply use the normal
insure command
insure -g -o strsort strsort.c
In addition to compiling the program, this will create a file called
tca.map which describes the layout
of your source file. We can now perform a set of tests on the application.
A few samples are shown in Figure 2. The
statements beginning with the $ symbols are the
commands executed and the remaining text is the response of the system.
$ strsort -a aaaa bbbb
aaaa bbbb
** TCA log data will be merged with tca.log **
$ strsort -a bbb aaa
aaa bbb
** TCA log data will be merged with tca.log **
$strsort -d aaa bbbb
bbbb aaa
** TCA log data will be merged with tca.log **
$ strsort -d bbb aaa
bbb aaa
** TCA log data will be merged with tca.log **
Figure 2. Sample test runs of the string sorting program
Note the following features from the figure.
Analysis of the test coverage data is performed using the
tca command. There are several summary levels. The
ones we will introduce here are:
- Overall summary (
default )
- Shows the percentage coverage at the application
level - i.e., summed over all program entities.
- Function summary (
-df switch)
- Displays the coverage of each individual function in
the application.
- Source code summary (
-ds switch)
- Displays the source code with each block marked as executed
(.) or not (!).
To see these commands in action, execute the command
tca tca.log
This displays the top level summary, shown below.
COVERAGE SUMMARY
================
1 block untested
12 blocks tested
92% covered
As can be seen, the overall coverage is quite high. However, one program
block remains untested. To find out which one is untested, execute
the command
tca -df tca.log
This command displays the function level summary, and includes functions
that are 100% tested.
COVERAGE SUMMARY - by function
================
blocks blocks %cov = functions sorted
untested tested %tested by name
------------------------------------------------------
0 4 100% bubble_sort [strsort.c, line 7-13]
1 8 88% main [strsort.c, line 26-45]
From this listing, we can see that the function bubble_sort
is completely tested, but that one block in main remains
untested. To find out which one, we execute the command
tca -ds tca.log
This results in the following output.
UNTESTED BLOCKS - by file
===============
FILE strsort.c 92% covered: 1 untested / 12 tested
/*
* File: strsort.c
*/
...
main(argc, argv)
int argc;
char **argv;
{
int i, dir, length, start;
. -> if (argc > 1 && argv[1][0] == `-`){
. -> if (argv[1][1] == `a`){
. -> dir = 1; length = argc-2; start = 2;
. -> } else if (argv[1][1] == `d`){
. -> dir = -1; length = argc-2; start = 2;
}
} else {
! -> dir = 1; length = argc; start = 1;
}
. -> bubble_sort(argv+start, length, dir);
for (i = 2; i < argc; i++)
. -> printf("%s ", argv[i]);
. -> printf("\n");
return 0;
}
...
This listing shows exactly which statements have been executed and which
have not. We can see from the results that the untested block corresponds
to the case where strsort is executed with neither
the -a nor -d command line
switches.
The previous analysis tells us that we can achieve 100% test coverage in
this example by executing the strsort program without
either of its two switches.
To complete testing, therefore, we can execute the program with the command
strsort aaa bbb
This produces the following output
[strsort.c:15] **READ_NULL**
>> if(dir * strcmp(a[j-1], a[j]) > 0) {
Reading null pointer: a[j]
Stack trace where the error occurred:
bubble_sort() strsort.c, 15
main() strsort.c, 41
**Memory corrupted. Program may crash!!**
** TCA log data will be merged with tca.log **
which indicates that Insure++ has found an error in this code
block.(Footnote 1)
However, if we now execute the command tca tca.log ,
we see the following output
COVERAGE SUMMARY
================
0 blocks untested
13 blocks tested
100% covered
which indicates that the application has now been 100% tested.
This means that the set of test cases that we have now run, including this
last one, completely exercises all the code in this application. It
would now make sense to incorporate these test cases (in conjunction
with Insure++) into a quality assurance test suite for this code.
There are several .psrc options you can use to
control coverage analysis. These are documented in the
Configuration Files section.
If you are using Insure++, coverage analysis information will be
automatically built into your program. At any time after you have run
your code you can use the tca command to find any
blocks which have not been executed. For clarity, the process is broken
down into three steps below, along with important information about
what's happening at each stage.
- Compiling applications and building their coverage analysis
database (usually named "
tca.map ").
- Running test cases against applications that have been compiled
with coverage analysis enabled, which creates entries in the
TCA log file (usually named
"
tca.log ").
- Running the TCA analysis tool to generate coverage
analysis reports from the given coverage log file(s).
You can make your program displays a coverage
analysis report when it exits by adding the
insure++.summarize coverage
option to your .psrc file. The
coverage_switches option lets you set flags to control
the output just as though you were passing those switches to
tca .
At compile time, Insure++ creates a database of each file
processed, describing how many blocks and statements are in each
file and function. This database is called a map file, because it
provides TCA with a map of how your program is laid out.
By default, the name of this file is
tca.map , but you can change the name of this file
by adding a
coverage_map_file
option to your .psrc file.
Ideally, all the files in your application should store their information
in the same map file. If your source code is spread across several
directories, you will probably want to set the map filename using a
full path, e.g.
insure++.coverage_map_file ~/project.map
If you compile several files simultaneously and they are all trying to
modify the same map file, you may end up with a corrupt map file. In
this case, you will need to delete the original map file and recompile
the application you are interested in.
As mentioned earlier, if you are only interested in
coverage information and not debugging, you can add the
-Zcov option to the insure
command lines that build your program. Remember to use
-Zcov consistently, i.e. at both compilation and
linking, if you use it at all.
At runtime, your program (compiled with Insure++) writes a log
file, which records the blocks that were actually executed during a
specific run. By default, this file is called
tca.log , but as with the map file, you can change
the name of this file by adding a coverage_log_file option to your
.psrc file. Normally, each time you run your
program the new log information will be combined with any found in
the existing log file, unless the data is not compatible (because you
changed your code and recompiled, for example).
Another useful option is to generate a new log file each time your
application runs. You can do this by taking advantage of the
%n filename option, for example
insure++.coverage_log_file tca.log.%n
In this example, each run would make a new file, such as
tca.log.0 , tca.log.1 , and so forth.
If your program forks, you will need to use this option so that each
child creates its own log file.
After you have created one or more log files, you can use the
tca command to get the information in which you
are interested. tca normally sends its reports to
stdout . If you would like to use the graphical version
to generate coverage reports, please see
page 23.
You can specify any number of log files on the command line, and TCA
will combine the data before displaying the results. (If the log files are
not compatible, e.g. because they are from different applications,
TCA will throw out the ones which don't match the first log file).
TCA will also need to read the map file created at compilation
time. Since this name is stored in the log file, you won't normally
need to specify it.
By default, TCA will give you a quick summary of how much of
your code was tested. Using different options, you can get detailed reports
of coverage by file, function, or even block.
For each block, TCA can tell you how many times it was executed,
summing over all the log files (unless coverage_boolean was on at compile time, the default
setting).
Note that if a single statement spans several lines of source code,
TCA treats the statement as lying on the last line - this is
only important for understanding the output of TCA, and does not
effect how coverage statistics are calculated.
Unlike some other coverage analysis tools which work on a
line-by-line basis, TCA is able to group your code into
logical blocks. A block is a group of statements which must always
be executed as a group. For example, the following code has three
statements, but only one block.
i = 3;
j = i+3;
return i+j;
Some of the advantages of using blocks over lines are
- lines of code which have several blocks are treated separately
- grouping equivalent statements into a single block reduces the
amount of data you need to analyze
- By treating labels as a separate group, you can actually detect
which paths have been executed in addition to which statements.
Note that conditional expressions containing &&
or || are grouped with the statement they are part
of. Also, the three elements of a for loop are treated
as part of the for statement (e.g.
e1 , e2 , and e3
in the code fragment for(e1;e2;e3) ).
Figures 3,
4, and 5
illustrate the above concepts with a simple test program.
Figure 3 shows the original code
and how the blocks are determined. In this particular example, there
are 16 blocks.
Figure 3. Sample program divided into blocks
Figure 4 shows the output of
"tca -ct -ds tca.log" after several test runs
with different values (test ; test 2 ; test 2 ; test 4 ; test 7
; test 3 ). By looking at this, you can see which paths have
been executed and which have not. Notice that counts are only given at the
beginning of each block, and not for each statement within
each block.
BLOCK USAGE - by file
============
FILE test.c 87% covered: 2 untested / 14 tested
#include <ctype.h>
main(int argc, char **argv) {
int flag;
6 -> if (argc &l;t 2 || !isdigit(argv[1][0])) {
1 -> printf(\`Bad argument(s)\n\');
exit(1);
}
5 -> switch(atoi(argv[1])) {
0,2,1 -> case 1: case 2: case 3:
3 -> flag = 1;
break;
1 -> case 4:
0 -> case 5:
1 -> flag = 2;
break;
1 -> default:
1 -> flag = 0;
break;
}
5,4,1 -> if (flag > 0) flag = 1; else flag = 0;
5 -> printf(\`Flag is %s\n\', flag ? \`1\' : \`0\');
exit(0);
}
Figure 4. Output from "tca -ct -ds tca.log "
Finally, Figure 5 shows the terser
output of "tca -ds tca.log" . In this instance,
only blocks which have not been executed are marked (corresponding to
blocks with a count of zero in the previous figure). The "!"
character symbolizes not executed. For lines with multiple
blocks, you will also see the "." character which means that
group was executed. This is so you can easily identify which blocks on that
line were not tested. Once again, only the first line of code within each
block will be marked in this fashion. Of you are using the graphical
TCA, the first line of the block will be colored as executed
(red) while the rest of the block will be colored as not executed
(black).
UNTESTED BLOCKS - by file
===============
FILE test.c 87% covered: 2 untested / 14 tested
#include <ctype.h>
main(int argc, char **argv) {
int flag;
. -> if (argc < 2 || !isdigit(argv[1][0])) {
. -> printf(\`Bad argument(s)\n\');
exit(1);
}
. -> switch(atoi(argv[1])) {
!.. -> case 1: case 2: case 3:
. -> flag = 1;
break;
. -> case 4:
! -> case 5:
. -> flag = 2;
break;
. -> default:
. -> flag = 0;
break;
}
... -> if (flag > 0) flag = 1; else flag = 0;
. -> printf(\`Flag is %s\n\', flag ? \`1\' : \`0\');
exit(0);
}
Figure 5. Output from "tca -ds tca.log "
The TCA display is a graphical representation of the reports
generated during run time. By utilizing this tool, you will be able
to view your tca.log files with ease. Much like
Insra, TCA
allows you to load and save files, browse through source code, and
even access on-line help.
Figure 6. Sample TCA display
The TCA display may be invoked by calling tca
with the -X switch along with any other command line
options. The following subset of TCA command line options are
meaningful for the graphical tool, while the remaining unsupported ones
are silently ignored.
Command line options:
Command
|
Explanation
|
-df
|
Display by function
|
-do
|
Display by object/class
|
-dF
|
Display by file
|
-dd
|
Display by directory
|
-ns
|
Simple function names
|
-ne
|
Extended function name - include argument types
|
-nm
|
Include modifiers const/volatile in function arguments
|
-ff name
|
Only show coverage related to function "name". This option only
applies when -df or -dF is specified.
|
-fo name
|
Only show coverage related to object "name". This option only applies
when -do is specified.
|
-fF name
|
Only show coverage related to file "name". "name&qout must
include the full path to the file. This option only applies when
-dF -df is specified.
|
-fd name
|
Only show coverage related to directory "name". This option only
applies when -dd is specified.
|
-s {keys}
|
Sort output by keys (d,F,n,%,#,b,1 )
|
-ct
|
Show hit counts in the source browser
|
By default, TCA displays a report based on the log files that
were included in the command line when the program was started.
Coverage statistics from additional files may be included in the report
by clicking on the Load button and selecting a new log file.
The data contained in the newly selected log file will be combined with
the existing data and a new report will be generated.
The Browse button generates a new windows containing the next level
of coverage detail. For example, if you are currently displaying a report
"by directory", clicking Browse will open a new window
displaying a report "by file" for that directory. If you were to
click Browse again, you would get another new window showing a report
"by function" for the file(s). Clicking Browse a final time
will display the actual source code itself, annotated with coverage
information for each block.
Double-clicking on a line in the display is the equivalent to selecting
a line and clicking the Browse button.
Figure 7. Browsing Annotated Source Code.
The level of detail displayed may be changed by clicking on the
Reports button. A dialog box will appear, allowing you to choose
from one of four report types: by directory, by file,
by function, or by class.
The order in which the coverage information is presented may be
modified by clicking on the Sort button. A dialog box will appear
in which you can enter the sort keys to be used. Any combination of the
following keys may be used:
Sort Keys
|
Explanation
|
d
|
by directory
|
F
|
by file name
|
f
|
by function
|
%
|
by percent covered
|
#
|
by number of hits
|
b
|
by number of blocks
|
For example, in order to sort by percent covered and decide collisions
by the function name, the sort key string should be
"%f" .
The current sort key string is displayed on the status bar.
This button becomes active when TCA cannot perform a given task.
Pressing it will bring up a window that describes the error(s).
Clicking on the Help button invokes context-sensitive help for TCA.
The shape of the cursor will change to a question mark/arrow combination.
Clicking on a region of the display will then cause the Help window
to appear with a description of the relevant features.
Now that you have all this coverage analysis information, what's the
best way to use it? Typically, you will have several tests for your
code designed to exercise various features. Together, these tests make
a test suite. After you have run your tests, use the tca
command to discover which blocks have not been executed. This will
indicate deficiencies in your test suite.
At this point, you should create more tests to try and exercise the
code that was missed by your test suite so far. After you have
created more tests, you can repeat the process. If the goal of 100%
coverage is unreachable, you will need to make a subjective decision
about how thorough you can afford to be.
Figure 8. An iterative approach to building a test suite.
Footnotes
- (1)
- Finding and fixing this error is left as an exercise for the
reader. The reader should also remember that if she built the
program with the
-Zcov option, this bug
would not have been detected by Insure++.
|