Testing is an integral part of the software development process for embedded systems and its necessity to attain software quality is undisputed. Suprisingly, only a few tools are available to assist the software developer with this mostly unpopular activity. This particularly applies to the area of dynamic unit testing.

The dynamic function testing of embedded software is usually performed manually with human interaction. The program to be tested is loaded into a simulator or onto the actual target system, perhaps with the help of a JTAG debugger or an-circuit emulator, the latter of which would allow testing to take place in absence of the target system.

During manual testing, the program under test is executed step by step and it is closely observed to ensure its behavior is as expected. In order to perform the same test with other values, the debugger can be used to set these values in the target system’s memory before the function under test is called. New program sections are often implemented, particularly those written for the test, in order to make certain inputs available to the function. This practice is tedious and also error-prone, since it is easy for the human tester to overlook any undesired results. Above all, once a manual test is performed, the input values used and the corresponding result are usually no longer available for use in subsequent test repetitions.

Sophisticated debuggers often allow a previous test’s input values to be extracted from a file. If this is the case, it’s usually possible to use the debugger to compare a function’s test result with values stored in the file and thus recognize if the test resulted in an error or not. But regardless of how user friendly the debugger is, this method is always complex. In principle, a database with input and result values should be created, where the number, order and above all the data types of these values are stored. It’s also necessary to specify how the debugger accesses the database, how a test result is determined, how a test evaluation is generated, how tests are reproduced, etc. This would eventually lead to the creation of a custom test environment.

Such a test environment is usually created for a particular function under test and it is therefore inflexible. This is then a setback if the number, order or data type of the function's parameters change. Adapting and upgrading a function’s test environment is often as complex as modifying the function itself. A test environment that facilitates the easy efficient repetition of all tests is invaluable since if the implementation is even slightly modified, the new testing of all functions with all test cases must take place from scratch. These reiterations are termed regression tests and due to their complexity, they are almost impossible to perform as part of manual interactive testing.

In practice, proprietary company-internal test environments already exist and they provide a solution to the issues mentioned above. The quality and flexibility of these test environments varies according to the amount of development effort invested into them. The reason why no standardized solutions are used for the development of embedded software is simple: surprisingly there are hardly any commercially available tools whose functionality is aimed at providing a solution to the problems mentioned.

Tessy is a commercially available tool for the automatic dynamic unit testing of software written in C for embedded systems. In order to begin testing with Tessy, the user simply selects the software module that contains the unit (in this case a C-function) to be tested. Tessy analyses this module and lists all C functions it finds within it. The user then selects the function to be tested.

Tessy analyses the source code of the function and determines which variables of the function interface are inputs, outputs (result) and/or both. Tessy’s internal Test Interface Editor (TIE) graphically displays the number, type and passing direction (Input, Output, InOut) of function interface variables, which can if necessary be manually modified or completed by the user.

Figure 1: The Test Interface Editor (TIE) displays the number, type and passing direction of interface variables

Tessy then automatically generates additional source code for the so-called test driver, which calls the function to be tested. If necessary, Tessy can also generate code for placeholders, which are used as substitutes for functions or external variables that are not yet implemented. Tessy then generates the test application using the start up code of the relevant microcontroller, the test driver, the optional placeholders and the function under test. In doing so, a cross compiler for the relevant target system is usually used.

Test case input values and the values of expected test case results can now be specified in Tessy using its integrated Test Data Editor (TDE). The TDE graphically displays the interface of the function under test and also allows the viewing of objects ranging from complex interface elements such as structures to elementary data types. Although the inputting of test data is a major task for the user, Tessy’s TDE allows this to be accomplished with particular ease and efficiency. The TDE can also be used to specify the expected results of a test case. Values are automatically stored in a database

Figure 2: Test case input and output values are specified using the Test Data Editor (TDE)

In order to run a test, Tessy loads the test application via a debugger into the test system, which could be an in-circuit emulator. The test thus makes use of the actual microcontroller and also checks the cross compiler used. Tessy then runs all test cases in sequence on the test system and checks if the actual result of each test case executed corresponds with the expected result. Test values are extracted from the database one by one and are not included in the test application. This allows the size of a test application to be independent of the number of test cases. Hence in principle, the number of test cases is unlimited and Tessy can be used with 8-bit microcontrollers that have limited memory.

Tessy generates reports on the execution and results of test cases in different degrees of detail.

Figure 3: Test Evaluation – Test cases yielding an unexpected result are marked in red

Of course, if the execution of a test case with a particular set of input values did not yield the expected result, the causes of this failure need to be determined. The tight integration of Tessy with debuggers such as Hitex’s HiTOP enable Tessy to re-run the test case, whereby Tessy sets a breakpoint at the entry point of the function to be tested. Test case execution is thus conveniently transferred to the debugger where it continues at the very beginning of the function to be tested. The function is called with the exact data that caused the unexpected result to occur. This allows the error detection process to begin immediately. As soon as the error is found, Tessy’s integral editor can be used to correct the function’s source code and all tests can then easily be re-run using the newly modified test object.

Regression tests are re-runs of successfully completed test cases. They provide verification that the modifications and enhancements made to a program are not the cause of undesired effects. Automatic regression tests are particularly useful after optimizing software or whenever a new compiler version is used. Tessy includes a batch function that enables extensive regression testing to take place without any user intervention.

Further development of a program can sometimes cause test cases to become un-executable, since the interface of the function under test has changed, for example due the introduction of an additional parameter. With self-created test environments, a large amount of adaptation is usually necessary. Tessy however completes this adaptation comfortably and to a large extent automatically.

In order for test cases to be updateable and thus reusable at any time, test data, including data for modified interfaces must be reusable in the most efficient and extensive means possible. The re-running of tests implies regression testing and regression testing is an essential software quality assurance procedure. Only with regression tests is it possible to optimize software in terms of size, performance, maintainability and reusability, since regression tests guarantee the actual functionality remains unaltered.

Test Coverage Analysis indicates the extent to which testing has covered the software under test. If a certain percentage of coverage is lacking, it could be that the tests were insufficient or some other abnormality exists such as "dead code.”

Tessy also determines the paths of program flow covered when running a test. By comparing the number of paths covered to the number of paths possible, Tessy is able to reveal the Path Coverage, also known as C1 Coverage. Tessy also indicates the paths covered by individual test cases: if the Path Coverage was insufficient, it’s easy to recognize the paths that were not tested.
The Classification Tree Editor (CTE) is an integral part of Tessy and it is used to specify test cases based on the Classification Tree Method. The Classification Tree Method is a systematic means of determining a set of low-redundancy, error sensitive test cases and it leads from a problem specification to actual test case specifications. The so-called classification tree is created during this process.

Use of the Classification Tree Method requires careful consideration of specifications and test cases. This allows insight to be gained into issues such as:

• What aspects are available for the test and how are they to be covered by test cases?
• What number of test cases is sufficient?
• Are there any redundant test cases?
• What is the expected amount of testing required?

The CTE is a syntax controlled graphical editor used to create a classification tree and to determine the test case specifications that are based on this tree. The CTE manages test case specifications, test cases and test case remarks, etc. and it can provide this information to other tools, as well as to Tessy.

Figure 4: The Classification Tree Editor is used to draw the classification tree and to specify test cases

Tessy greatly improves the ease and efficiency of testing applications and to a large extent it replaces proprietary test environments. At the beginning of a test, Tessy generates any components still missing such as test drivers or placeholder functions. The tool’s test data management allows all test data to be conserved for use in subsequent tests. The integration of Tessy with debuggers allows efficient error detection whenever test cases yield unexpected results. If a program is modified, Tessy can be used to correspondingly update and adapt the program’s tests. Tessy is thus particularly suited to regression testing.

The planning of test cases at the beginning of development ties in with the test coverage analysis at the end of testing.

Tessy and the CTE are available from Hitex Development Tools.


Author:
Frank Büchner

Contact
Hitex Development Tools, Greschbachstrasse 12, D-76229 Karlsruhe / Germany
www.hitex.de / www.hitex.us