Documentation: Add proposal for firmware unit testing
Signed-off-by: Jan Dabros <jsd@semihalf.com> Change-Id: I552d6c3373219978b8e5fd4304f993d920425431 Reviewed-on: https://review.coreboot.org/c/coreboot/+/39893 Tested-by: build bot (Jenkins) <no-reply@coreboot.org> Reviewed-by: Paul Fagerburg <pfagerburg@chromium.org> Reviewed-by: Julius Werner <jwerner@chromium.org>
This commit is contained in:
Jan Dabros
committed by
Patrick Georgi
Patrick Georgi
parent
da5e07e6c7
commit
6449b67427
319
Documentation/technotes/2020-03-unit-testing-coreboot.md
Normal file
319
Documentation/technotes/2020-03-unit-testing-coreboot.md
Normal file
@@ -0,0 +1,319 @@
|
|||||||
|
# Unit testing coreboot
|
||||||
|
|
||||||
|
## Preface
|
||||||
|
First part of this document, Introduction, comprises disambiguation for what
|
||||||
|
unit testing is and what is not. This definition will be a basis for the whole
|
||||||
|
paper.
|
||||||
|
|
||||||
|
Next, Rationale, explains why to use unit testing and how coreboot specifically
|
||||||
|
may benefit from it.
|
||||||
|
|
||||||
|
This is followed by evaluation of different available free C unit test
|
||||||
|
frameworks. Firstly, collection of requirements is provided. Secondly, there is
|
||||||
|
a description of a few selected candidates. Finally, requirements are applied to
|
||||||
|
candidates to see if they might be a good fit.
|
||||||
|
|
||||||
|
Fourth part is a summary of evaluation, with proposal of unit test framework
|
||||||
|
for coreboot to be used.
|
||||||
|
|
||||||
|
Finally, Implementation proposal paragraph touches how build system and coreboot
|
||||||
|
codebase in general should be organized, in order to support unit testing. This
|
||||||
|
comprises couple of design considerations which need to be addressed.
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
A unit test is supposed to test a single unit of code in isolation. In C
|
||||||
|
language (in contrary to OOP) unit usually means a function. One may also
|
||||||
|
consider unit under test to be a single compilation unit which exposes some
|
||||||
|
API (set of functions). A function, talking to some external component can be
|
||||||
|
tested if this component can be mocked out.
|
||||||
|
|
||||||
|
In other words (looking from C compilation angle), there should be no extra
|
||||||
|
dependencies (executables) required beside unit under test and test harness in
|
||||||
|
order to compile unit test binary. Test harness, beside code examining a
|
||||||
|
routines, may comprise test framework implementation.
|
||||||
|
|
||||||
|
It is hard to apply this strict definition of unit test to firmware code in
|
||||||
|
practice, mostly due to constraints on speed of execution and size of final
|
||||||
|
executable. coreboot codebase often cannot be adjusted to be testable. Because
|
||||||
|
of this, coreboot unit testing subsystem should allow to include some additional
|
||||||
|
source object files beside unit under test. That being said, the default and
|
||||||
|
goal wherever possible, should be to isolate unit under test from other parts.
|
||||||
|
|
||||||
|
Unit testing is not an integration testing and it doesn't replace it. First of
|
||||||
|
all, integration tests cover larger set of components and interactions between
|
||||||
|
them. Positive integration test result gives more confidence than a positive
|
||||||
|
unit test does. Furthermore, unit tests are running on the build machine, while
|
||||||
|
integration tests usually are executed on the target (or simulator).
|
||||||
|
|
||||||
|
## Rationale
|
||||||
|
Considering above, what is the benefit of unit testing, especially keeping in
|
||||||
|
mind that coreboot is low-level firmware? Unit tests should be quick, thus may
|
||||||
|
be executed frequently during development process. It is much easier to build
|
||||||
|
and run a unit test on a build machine, than any integration test. This in turn
|
||||||
|
may be used by dev to gather extra confidence early during code development
|
||||||
|
process. Actually developer may even write unit tests earlier than the code -
|
||||||
|
see [TDD](https://en.wikipedia.org/wiki/Test-driven_development) concept.
|
||||||
|
|
||||||
|
That being said, unit testing embedded C code is a difficult task, due to
|
||||||
|
significant amount of dependencies on underlying hardware. Mocking can handle
|
||||||
|
some hardware dependencies. However, complex mocks make the unit test
|
||||||
|
susceptible to failing and can require significant development effort.
|
||||||
|
|
||||||
|
Writing unit tests for a code (both new and currently existing) may be favorable
|
||||||
|
for the code quality. It is not only about finding bugs, but in general - easily
|
||||||
|
testable code is a good code.
|
||||||
|
|
||||||
|
coreboot benefits the most from testing common libraries (lib/, commonlib/,
|
||||||
|
payloads/libpayload) and coreboot infrastructure (console/, device/, security/).
|
||||||
|
|
||||||
|
## Evaluation of unit testing frameworks
|
||||||
|
|
||||||
|
### Requirements
|
||||||
|
Requirements for unit testing frameworks:
|
||||||
|
|
||||||
|
* Easy to use
|
||||||
|
* Few dependencies
|
||||||
|
|
||||||
|
Standard C library is all we should need
|
||||||
|
|
||||||
|
* Isolation between tests
|
||||||
|
* Support for mocking
|
||||||
|
* Support for some machine parsable output
|
||||||
|
* Compiler similarity
|
||||||
|
|
||||||
|
Compiler for the host _must_ support the same language standards as the target
|
||||||
|
compiler. Ideally the same toolchain should be used for building firmware
|
||||||
|
executables and test binaries, however the host complier will be used to build
|
||||||
|
unit tests, whereas the coreboot toolchain will be used for building the
|
||||||
|
firmware executables. For some targets, the host compiler and the target
|
||||||
|
compiler could be the same, but this is not a requirement.
|
||||||
|
|
||||||
|
* Same language for tests and code
|
||||||
|
|
||||||
|
Unit tests will be written in C, because coreboot code is also written in C
|
||||||
|
|
||||||
|
### Desirables
|
||||||
|
|
||||||
|
* Easy to integrate with build system/build tools
|
||||||
|
|
||||||
|
Ideally JUnit-like XML output format for Jenkins
|
||||||
|
|
||||||
|
* Popularity is a plus
|
||||||
|
|
||||||
|
We want a larger community for a couple of reasons. Firstly, easier access to
|
||||||
|
people with knowledge and tutorials. Secondly, bug fixes for the top of tree
|
||||||
|
are more frequent and known issues are usually shorter in the pending state.
|
||||||
|
Last but not least, larger reviewer pool means better and easier upstream
|
||||||
|
improvements that we would like to submit.
|
||||||
|
|
||||||
|
* Extra features may be a plus
|
||||||
|
* Compatible license
|
||||||
|
|
||||||
|
This should not be a blocker, since test binaries are not distributed.
|
||||||
|
However ideally compatible with GPL.
|
||||||
|
|
||||||
|
* IDE integration
|
||||||
|
|
||||||
|
### Candidates
|
||||||
|
There is a lot of frameworks which allow unit testing C code
|
||||||
|
([list](https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C) from
|
||||||
|
Wikipedia). While not all of them were evaluated, because that would take an
|
||||||
|
excessive amount of time, couple of them were selected based on the good
|
||||||
|
opinions among C devs, popularity and fitting above criteria.
|
||||||
|
|
||||||
|
* [SputUnit](https://www.use-strict.de/sput-unit-testing/)
|
||||||
|
* [GoogleTest](https://github.com/google/googletest)
|
||||||
|
* [Cmocka](https://cmocka.org/)
|
||||||
|
* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling)
|
||||||
|
|
||||||
|
We looked at several other test frameworks, but decided not to do a full evaluation
|
||||||
|
for various reasons such as functionality, size of the developer community, or
|
||||||
|
compatibility.
|
||||||
|
|
||||||
|
### Evaluation
|
||||||
|
* [SputUnit](https://www.use-strict.de/sput-unit-testing/)
|
||||||
|
* Pros
|
||||||
|
* No dependencies, one header file to include - that’s all
|
||||||
|
* Pure C
|
||||||
|
* Very easy to use
|
||||||
|
* BSD license
|
||||||
|
* Cons
|
||||||
|
* Main repo doesn’t have support for generating JUnit XML reports for
|
||||||
|
Jenkins to consume - this feature is available only on the fork from
|
||||||
|
SputUnit called “Sput_report”. It makes it niche in a niche, so there are
|
||||||
|
some reservations whether support for this will be satisfactory
|
||||||
|
* No support for mocks
|
||||||
|
* Not too popular
|
||||||
|
* No automatic test registration
|
||||||
|
* [GoogleTest](https://github.com/google/googletest)
|
||||||
|
* Pros
|
||||||
|
* Automatic test registration
|
||||||
|
* Support for different output formats (including XML for Jenkins)
|
||||||
|
* Good support, widely used, the biggest and the most active community out
|
||||||
|
of all frameworks that were investigated
|
||||||
|
* Available as a package in the most common distributions
|
||||||
|
* Test fixtures easily available
|
||||||
|
* Well documented
|
||||||
|
* Easy to integrate with an IDE
|
||||||
|
* BSD license
|
||||||
|
* Cons
|
||||||
|
* Requires C++11 compiler
|
||||||
|
* To make most out of it (use GMock) C++ knowledge is required
|
||||||
|
* [Cmocka](https://cmocka.org/)
|
||||||
|
* Pros
|
||||||
|
* Self-contained, autonomous framework
|
||||||
|
* Pure C
|
||||||
|
* API is well documented
|
||||||
|
* Multiple output formats (including XML for Jenkins)
|
||||||
|
* Available as a package in the most common distributions
|
||||||
|
* Used in some popular open source projects (libssh, OpenVPN, Samba)
|
||||||
|
* Test fixtures available
|
||||||
|
* Support for exception handling
|
||||||
|
* Cons
|
||||||
|
* No automatic test registration
|
||||||
|
* It will require some effort to make it work from within an IDE
|
||||||
|
* Apache 2.0 license (not compatible with GPLv2)
|
||||||
|
* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling)
|
||||||
|
* Pros
|
||||||
|
* Pure C (Unity testing framework itself, not test runner)
|
||||||
|
* Support for different output formats (including XML for Jenkins)
|
||||||
|
* There are some (rather easy) hints how to use this from an IDE (e.g. Eclipse)
|
||||||
|
* MIT license
|
||||||
|
* Cons
|
||||||
|
* Test runner (Ceedling) is not written in C - uses Ruby
|
||||||
|
* Mocking/Exception handling functionalities are actually separate tools
|
||||||
|
* No automatic test registration
|
||||||
|
* Not too popular
|
||||||
|
|
||||||
|
### Summary & framework proposal
|
||||||
|
After research, we propose using the Cmocka unit test framework. Cmocka fulfills
|
||||||
|
all stated evaluation criteria. It is rather easy to use, doesn’t have extra
|
||||||
|
dependencies, written fully in C, allows for tests fixtures and some popular
|
||||||
|
open source projects already are using it. Cmocka also includes support for
|
||||||
|
mocks.
|
||||||
|
|
||||||
|
Cmocka's limitations, such as the lack of automatic test registration, are
|
||||||
|
considered minor issues that will require only minimal additional work from a
|
||||||
|
developer. At the same time, it may be worth to propose improvement to Cmocka
|
||||||
|
community or simply apply some extra wrapper with demanded functionality.
|
||||||
|
|
||||||
|
## Implementation
|
||||||
|
|
||||||
|
### Framework as a submodule or external package
|
||||||
|
Unit test frameworks may be either compiled from source (from a git submodule
|
||||||
|
under 3rdparty/) or pre-compiled as a package. The second option seems to be
|
||||||
|
easier to maintain, while at the same time may bring some unwanted consequences
|
||||||
|
(different version across distributions, frequent changes in API). It makes sense
|
||||||
|
to initially experiment with packages and check how it works. If this will
|
||||||
|
cause any issues, then it is always possible to switch to submodule approach.
|
||||||
|
|
||||||
|
### Integration with build system
|
||||||
|
To get the most out of unit testing framework, it should be integrated with
|
||||||
|
Jenkins automation server. Verification of all unit tests for new changes may
|
||||||
|
improve code reliability to some extent.
|
||||||
|
|
||||||
|
### Build configuration (Kconfig)
|
||||||
|
While building unit under test object file, it is necessary to apply some
|
||||||
|
configuration (config) just like when building usual firmware. For simplicity,
|
||||||
|
there will be one default tests .config `qemu_x86_i440fx` for all unit tests. At
|
||||||
|
the same time, some tests may require running with different values of particular
|
||||||
|
config. This should be handled by adding extra header, included after config.h.
|
||||||
|
This header will comprise #undef of old CONFIG values and #define of the
|
||||||
|
required value. When unit testing will be integrated with Jenkins, it may be
|
||||||
|
preferred to use every available config for periodic builds.
|
||||||
|
|
||||||
|
### Directory structure
|
||||||
|
Tests should be kept separate from the code, while at the same time it must be
|
||||||
|
easy to match code with test harness.
|
||||||
|
|
||||||
|
We create new directory for test files ($(toplevel)/tests/) and mimic the
|
||||||
|
structure of src/ directory.
|
||||||
|
|
||||||
|
Test object files (test harness, unit under tests and any additional executables
|
||||||
|
are stored under build/tests/<test_name> directory.
|
||||||
|
|
||||||
|
Below example shows how directory structure is organized for the two test cases:
|
||||||
|
tests/lib/string-test and tests/device/i2c-test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
├── src
|
||||||
|
│ ├── lib
|
||||||
|
│ │ ├── string.c <- unit under test
|
||||||
|
│ │
|
||||||
|
│ ├── device
|
||||||
|
│ ├── i2c.c
|
||||||
|
│
|
||||||
|
├── tests
|
||||||
|
│ ├── include
|
||||||
|
│ │ ├── mocks <- mock headers, which replace original headers
|
||||||
|
│ │
|
||||||
|
│ ├── Makefile.inc <- top Makefile for unit tests subsystem
|
||||||
|
│ ├── lib
|
||||||
|
│ │ ├── Makefile.inc
|
||||||
|
│ │ ├── string-test.c <- test code for src/lib/string.c
|
||||||
|
│ │ │
|
||||||
|
│ ├── device
|
||||||
|
│ │ ├── Makefile.inc
|
||||||
|
│ ├── i2c-test.c
|
||||||
|
│
|
||||||
|
├── build
|
||||||
|
│ ├── tests <-all test-related executables
|
||||||
|
├── config.h <- default config used for tests builds
|
||||||
|
├── lib
|
||||||
|
│ ├── string-test <- all string-test executables
|
||||||
|
│ │ ├── run <- final test binary
|
||||||
|
│ │ ├── tests <- all test harness executables
|
||||||
|
│ │ ├── lib
|
||||||
|
│ │ ├── string-test.o <-test harness executable
|
||||||
|
│ │ ├── src <- unit under test and other src executables
|
||||||
|
│ │ ├── lib
|
||||||
|
│ │ ├── string.o <- unit under test executable
|
||||||
|
├── device
|
||||||
|
├── i2c-test
|
||||||
|
├── run
|
||||||
|
├── tests
|
||||||
|
│ ├── device
|
||||||
|
│ ├── i2c-test.o
|
||||||
|
├── src
|
||||||
|
├── device
|
||||||
|
├── i2c.o
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding new tests
|
||||||
|
For purpose of this description, let's assume that we want to add a new unit test
|
||||||
|
for src/device/i2c.c module. Since this module is rather simple, it will be enough
|
||||||
|
to have only one test module.
|
||||||
|
|
||||||
|
Firstly (assuming there is no tests/device/Makefile.inc file) we need to create
|
||||||
|
Makefile.inc in main unit test module directory. Inside this Makefile.inc, one
|
||||||
|
need to register new test and can specify multiple different attributes for it.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Register new test, by adding its name to tests variable
|
||||||
|
tests-y += i2c-test
|
||||||
|
|
||||||
|
# All attributes are defined by <test_name>-<attribute> variables
|
||||||
|
# <test_name>-srcs is used to register all input files (test harness, unit under
|
||||||
|
# test and others) for this particular test. Remember to add relative paths.
|
||||||
|
i2c-test-srcs += tests/device/i2c-test.c
|
||||||
|
i2c-test-srcs += src/device/i2c.c
|
||||||
|
|
||||||
|
# We can define extra cflags for this particular test
|
||||||
|
i2c-test-cflags += -DSOME_DEFINE=1
|
||||||
|
|
||||||
|
# For mocking out external dependencies (functions which cannot be resolved by
|
||||||
|
# linker), it is possible to register a mock function. To register new mock, it
|
||||||
|
# is enough to add function-to-be-mocked name to <test_name>-mocks variable.
|
||||||
|
i2c-test-mocks += platform_i2c_transfer
|
||||||
|
|
||||||
|
# Similar to coreboot concept, unit tests also runs in the context of stages.
|
||||||
|
# By default all unit tests are compiled to be ramstage executables. If one want
|
||||||
|
# to overwrite this setting, there is <test_name>-stage variable available.
|
||||||
|
i2c-test-stage:= bootblock
|
||||||
|
```
|
||||||
|
|
||||||
|
### Writing new tests
|
||||||
|
Full description of how to write unit tests and Cmocka API description is out of
|
||||||
|
the scope of this document. There are other documents related to this
|
||||||
|
[Cmocka API](https://api.cmocka.org/) and
|
||||||
|
[Mocks](https://lwn.net/Articles/558106/).
|
||||||
@@ -2,3 +2,4 @@
|
|||||||
|
|
||||||
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md)
|
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md)
|
||||||
* [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md)
|
* [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md)
|
||||||
|
* [Unit testing coreboot](2020-03-unit-testing-coreboot.md)
|
||||||
|
|||||||
Reference in New Issue
Block a user