Software development - a lot more than programming: Part 3

The manual source code review activity can be extended with static source code analysis. This is the process where a software tool analyses the source code of an application and automatically detects potential bugs or other types of problems in the source code.

Most tools that perform static source code analysis check the coding style versus a formal coding standard (the most popular one in the embedded industry is currently MISRA-C:2004). The coding standard typically limits the programmer’s flexibility and only allows using source code constructs that promote safety, reliability, maintenance and portability.

Another important feature of some static source code analysis tool is the capability to provide code metrics, which essentially is statistics about the source code. Code metrics can, for example, include the percentage of lines that contain comments, or information about the complexity level of each C/C++ function in the project. Overly complex functions should be rewritten into a simpler coding style to reduce the risk of bugs and to simplify maintenance.

Static source code analysis should be integrated in the C/C++ IDE to simplify its daily routine use, as is the case with the TrueINSPECTOR tool that integrates into Atollic TrueSTUDIO. TrueINSPECTOR lists all detected coding standard violations, but also displays graphical charts providing overview information, as well as providing detailed information on each coding rule and displaying bad and good code examples as a teaching aid as well.

A common misconception is that a static source code analysis tool is only needed at the end of the project, where some violating code lines can be fixed at the end of the project.

In fact, nothing is further from the truth. No one will start to rewrite code with thousands or perhaps tens of thousands of rule violations when the code is completed.

The correct approach is to use this type of tool at least daily, to ensure a gradual and iterative development where all code additions are checked and fixed as they are added.

Most embedded projects do not use a formal test methodology, although more and more teams look into the use of unit tests. Unit tests are essentially function calls into a C/C++ functions, where each function call uses a different combination of input parameter values, to drive different execution paths in the function.

But writing unit tests takes time, is boring and usually does not cover all the important execution paths anyway. Other problems are that development teams frequently have to focus on completing the code and no time is available to keep the unit tests up to date as the code changes during development. Source code and unit tests thus get out of sync and the unit tests become more or less useless.

Once the unit tests are written, additional problems arise regarding how to build, download and run them in a target board. There are several simple unit test tools for PC developers, but they rarely manage compilation, downloading and execution of the test suites in embedded boards.

A better approach is to use a full embedded test automation system. Such tools have not been common in the embedded industry previously, but new tools like Atollic TrueVERIFIER now bring test automation capabilities to embedded developers. An additional benefit is that it is fully integrated in the C/C++ IDE.

TrueVERIFIER analyses the source code of the application and auto-generates unit tests with important combinations of input parameter values to drive as many important combinations of execution paths in the function as possible.

Once the unit tests have been auto-generated, they are auto-compiled and auto-downloaded to the target board using the TrueSTUDIO compiler and debugger. Execution is then performed in the target board with dynamic execution flow analysis to measure the code coverage.

Once the test suite is completed, test results and code coverage information is uploaded to the IDE. TrueVERIFIER measures code coverage on MC/DC level, the test quality level required for flight control system software.

Once code is developed and tested, it is of importance to understand the quality of the testing being performed. It is of no use to conclude that testing is completed with successful test results, if the test procedures do not test more than a fraction of the software.

The majority of the software might still be untested, and any testing of those parts might well result in test failures, should they have been run!

Code coverage analysis (dynamic execution flow analysis) is commonly used to study what parts of the code have been tested, and hence measure the test quality. There are many different types of code coverage analysis, from very simple analysis up to very stringent types.

Code coverage analysis is often classified formally. The more advanced types of code coverage analysis (such as MC/DC described below) are often used for measuring test quality of safety critical software.

As an example, RTCA DO-178B (a standard for development of flight safety critical software) requires MC/DC testing of software on ‘Level-A criticality’, the most critical part of airborne software, where a software error can lead to a catastrophic situation with loss of aircraft or human lives.

Many projects also outside the aerospace industry would benefit from better control of what has been tested. In particular, this is valid for companies with high production volumes or products that are difficult to upgrade in the field.

The same goes for products where the supplier wants to keep its good reputation and where badwill can be costly for the company.

For these reasons, code coverage analysis ought to be used to verify whether the software has been tested well enough before delivery to customers.

Examples of different types of code coverage analysis are:

• Statement or block coverage: This type of code coverage only measures how many of the code lines or code blocks have been executed during a test session. It does not measure how branches in the execution flow affect which code lines or code blocks become executed.
• Function coverage: This type of code coverage only measures which or how many of the C/C++ functions have been called during a test session. It does not measure which or how many of the function calls in a code section is actually executed.
• Function call coverage: This type of code coverage also measures which or how many of the C/C++ function calls in a code section have actually been called during a test session.
• Branch coverage: This type of code coverage measures if all alternate branch paths have been executed in a code section (such as both the if- and the else- part in an if-statement, or that all cases have been executed in a switch-statement). Branch coverage typically requires a code section to be executed several or many times, as all alternative branch directions must be tested.
• Modified condition/decision coverage (MC/DC): This is a very advanced type of code coverage. It extends Branch coverage with the additional requirement that all subexpressions in complex decisions (such as in an if-statement) must drive the branch decision independently of the other subexpressions. This typically requires the code to be executed many times, with different combinations of values that fulfil many combinations of subexpression values.

Code coverage analysis on MC/DC-level is, of course, very complicated without sufficient tool support. There are, however, tools that do this automatically, such as TrueANALYZER.

The fact that the code coverage analysis is run in the target system is important. If it is done in a simulated environment on a PC, interaction with the target system (users push-buttons, other systems send communication packages etc) it cannot be done properly. Timing and code execution paths might become different, thus depreciating the value of the analysis.

The software industry has been progressing rapidly over the last few years, and developers of Windows software have very powerful tools at their disposal. At the same time, tools for development of embedded software are still more or less on the same level as years ago, and only offer features for editing, compiling and debugging for the most part.

It is time to give embedded system developers proper tools, in the form of highly integrated products that cover a much wider set of problems. Such tools are already available and will give development projects better possibilities to deliver well-designed software, on time and on budget, with improved quality.

See Part 1 and Part 2 of this series.