Docs/tutorial: Wrap the text in part?.md to 72 characters
Signed-off-by: Martin Roth <gaumless@gmail.com> Change-Id: I4f0a07b4ab729aafdb4a1149a7617cd34392cf12 Reviewed-on: https://review.coreboot.org/c/coreboot/+/64967 Tested-by: build bot (Jenkins) <no-reply@coreboot.org> Reviewed-by: Elyes Haouas <ehaouas@noos.fr>
This commit is contained in:
committed by
Martin L Roth
parent
25aeaac85b
commit
d2c3e26513
@@ -1,31 +1,32 @@
|
||||
# Writing unit tests for coreboot
|
||||
|
||||
## Introduction
|
||||
General thoughts about unit testing coreboot can be found in
|
||||
[Unit testing coreboot](../technotes/2020-03-unit-testing-coreboot.md).
|
||||
Additionally, [code coverage](../technotes/2021-05-code-coverage.md) support
|
||||
is available for unit tests.
|
||||
General thoughts about unit testing coreboot can be found in [Unit
|
||||
testing coreboot](../technotes/2020-03-unit-testing-coreboot.md).
|
||||
Additionally, [code coverage](../technotes/2021-05-code-coverage.md)
|
||||
support is available for unit tests.
|
||||
|
||||
This document aims to guide developers through the process of adding and writing
|
||||
unit tests for coreboot modules.
|
||||
This document aims to guide developers through the process of adding and
|
||||
writing unit tests for coreboot modules.
|
||||
|
||||
As an example of unit under test, `src/device/i2c.c` (referred hereafter as UUT
|
||||
"Unit Under Test") will be used. This is simple module, thus it should be easy
|
||||
for the reader to focus solely on the testing logic, without the need to spend
|
||||
too much time on digging deeply into the source code details and flow of
|
||||
operations. That being said, a good understanding of what the unit under test is
|
||||
doing is crucial for writing unit tests.
|
||||
As an example of unit under test, `src/device/i2c.c` (referred hereafter
|
||||
as UUT "Unit Under Test") will be used. This is simple module, thus it
|
||||
should be easy for the reader to focus solely on the testing logic,
|
||||
without the need to spend too much time on digging deeply into the
|
||||
source code details and flow of operations. That being said, a good
|
||||
understanding of what the unit under test is doing is crucial for
|
||||
writing unit tests.
|
||||
|
||||
This tutorial should also be helpful for developers who want to follow
|
||||
[TDD](https://en.wikipedia.org/wiki/Test-driven_development). Even though TDD
|
||||
has a different work flow of building tests first, followed by the code that
|
||||
satisfies them, the process of writing tests and adding them to the tree is the
|
||||
same.
|
||||
[TDD](https://en.wikipedia.org/wiki/Test-driven_development). Even
|
||||
though TDD has a different work flow of building tests first, followed
|
||||
by the code that satisfies them, the process of writing tests and adding
|
||||
them to the tree is the same.
|
||||
|
||||
## Analysis of unit under test
|
||||
First of all, it is necessary to precisely establish what we want to test in a
|
||||
particular module. Usually this will be an externally exposed API, which can be
|
||||
used by other modules.
|
||||
## Analysis of unit under test First of all, it is necessary to
|
||||
precisely establish what we want to test in a particular module. Usually
|
||||
this will be an externally exposed API, which can be used by other
|
||||
modules.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
@@ -34,66 +35,70 @@ used by other modules.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data,
|
||||
uint8_t mask, uint8_t shift)
|
||||
int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t data,
|
||||
uint8_t mask, uint8_t shift)
|
||||
int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg,
|
||||
uint8_t *data, uint8_t mask, uint8_t shift)
|
||||
int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg,
|
||||
uint8_t data, uint8_t mask, uint8_t shift)
|
||||
|
||||
For sake of simplicity, let's focus on `i2c_read_field` in this document.
|
||||
For sake of simplicity, let's focus on `i2c_read_field` in this
|
||||
document.
|
||||
```
|
||||
|
||||
Once the API is defined, the next question is __what__ this API is doing (or
|
||||
what it will be doing in case of TDD). In other words, what outputs we are
|
||||
expecting from particular functions, when providing particular input parameters.
|
||||
Once the API is defined, the next question is __what__ this API is doing
|
||||
(or what it will be doing in case of TDD). In other words, what outputs
|
||||
we are expecting from particular functions, when providing particular
|
||||
input parameters.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data,
|
||||
uint8_t mask, uint8_t shift)
|
||||
int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg,
|
||||
uint8_t *data, uint8_t mask, uint8_t shift)
|
||||
|
||||
This is a method which means to read content of register `reg` from i2c device
|
||||
on i2c `bus` and slave address `chip`, applying bit `mask` and offset `shift`
|
||||
to it. Returned data should be placed in `data`.
|
||||
This is a method which means to read content of register `reg` from
|
||||
i2c device on i2c `bus` and slave address `chip`, applying bit `mask`
|
||||
and offset `shift` to it. Returned data should be placed in `data`.
|
||||
```
|
||||
|
||||
The next step is to determine all external dependencies of UUT in order to mock
|
||||
them out. Usually we want to isolate the UUT as much as possible, so that the
|
||||
test result depends __only__ on the behavior of UUT and not on the other
|
||||
modules. While some software dependencies may be hard to be mock (for example
|
||||
due to complicated dependencies) and thus should be simply linked into the test
|
||||
binaries, all hardware dependencies need to be mocked out, since in the
|
||||
user-space host environment, targets hardware is not available.
|
||||
The next step is to determine all external dependencies of UUT in order
|
||||
to mock them out. Usually we want to isolate the UUT as much as
|
||||
possible, so that the test result depends __only__ on the behavior of
|
||||
UUT and not on the other modules. While some software dependencies may
|
||||
be hard to be mock (for example due to complicated dependencies) and
|
||||
thus should be simply linked into the test binaries, all hardware
|
||||
dependencies need to be mocked out, since in the user-space host
|
||||
environment, targets hardware is not available.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
`i2c_read_field` is calling `i2c_readb`, which eventually invokes
|
||||
`i2c_transfer`. This method simply calls `platform_i2c_transfer`. The last
|
||||
function in the chain is a hardware-touching one, and defined separately for
|
||||
different SOCs. It is responsible for issuing transactions on the i2c bus.
|
||||
For the purpose of writing unit test, we should mock this function.
|
||||
`i2c_transfer`. This method simply calls `platform_i2c_transfer`. The
|
||||
last function in the chain is a hardware-touching one, and defined
|
||||
separately for different SOCs. It is responsible for issuing
|
||||
transactions on the i2c bus. For the purpose of writing unit test,
|
||||
we should mock this function.
|
||||
```
|
||||
|
||||
## Adding new tests
|
||||
In order to keep the tree clean, the `tests/` directory should mimic the `src/`
|
||||
directory, so that test harness code is placed in a location corresponding to
|
||||
UUT. Furthermore, the naming convention is to add the suffix `-test` to the UUT
|
||||
name when creating a new test harness file.
|
||||
In order to keep the tree clean, the `tests/` directory should mimic the
|
||||
`src/` directory, so that test harness code is placed in a location
|
||||
corresponding to UUT. Furthermore, the naming convention is to add the
|
||||
suffix `-test` to the UUT name when creating a new test harness file.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
Considering that UUT is `src/device/i2c.c`, test file should be named
|
||||
`tests/device/i2c-test.c`. When adding a new test file, it needs to be
|
||||
registered with the coreboot unit testing infrastructure.
|
||||
`tests/device/i2c-test.c`. When adding a new test file, it needs to
|
||||
be registered with the coreboot unit testing infrastructure.
|
||||
```
|
||||
|
||||
Every directory under `tests/` should contain a Makefile.inc, similar to what
|
||||
can be seen under the `src/`. Register a new test in Makefile.inc, by
|
||||
__appending__ test name to the `tests-y` variable.
|
||||
Every directory under `tests/` should contain a Makefile.inc, similar to
|
||||
what can be seen under the `src/`. Register a new test in Makefile.inc,
|
||||
by __appending__ test name to the `tests-y` variable.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
@@ -103,10 +108,11 @@ __appending__ test name to the `tests-y` variable.
|
||||
tests-y += i2c-test
|
||||
```
|
||||
|
||||
Next step is to list all source files, which should be linked together in order
|
||||
to create test binary. Usually a tests requires only two files - UUT and test
|
||||
harness code, but sometimes more is needed to provide the test environment.
|
||||
Source files are registered in `<test_name>-srcs` variable.
|
||||
Next step is to list all source files, which should be linked together
|
||||
in order to create test binary. Usually a tests requires only two files
|
||||
- UUT and test harness code, but sometimes more is needed to provide the
|
||||
test environment. Source files are registered in `<test_name>-srcs`
|
||||
variable.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
@@ -117,9 +123,10 @@ Source files are registered in `<test_name>-srcs` variable.
|
||||
i2c-test-srcs += src/device/i2c.c
|
||||
```
|
||||
|
||||
Above minimal configuration is a basis for further work. One can try to build
|
||||
and run test binary either by invoking `make tests/<test_dir>/<test_name>` or by
|
||||
running all unit tests (whole suite) for coreboot `make unit-tests`.
|
||||
Above minimal configuration is a basis for further work. One can try to
|
||||
build and run test binary either by invoking `make
|
||||
tests/<test_dir>/<test_name>` or by running all unit tests (whole suite)
|
||||
for coreboot `make unit-tests`.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
@@ -135,31 +142,34 @@ running all unit tests (whole suite) for coreboot `make unit-tests`.
|
||||
make unit-tests
|
||||
```
|
||||
|
||||
When trying to build test binary, one can often see linker complains about
|
||||
`undefined reference` to couple of symbols. This is one of solutions to
|
||||
determine all external dependencies of UUT - iteratively build test and resolve
|
||||
errors one by one. At this step, developer should decide either it's better to
|
||||
add an extra module to provide necessary definitions or rather mock such
|
||||
dependency. Quick guide through adding mocks is provided later in this doc.
|
||||
When trying to build test binary, one can often see linker complains
|
||||
about `undefined reference` to couple of symbols. This is one of
|
||||
solutions to determine all external dependencies of UUT - iteratively
|
||||
build test and resolve errors one by one. At this step, developer should
|
||||
decide either it's better to add an extra module to provide necessary
|
||||
definitions or rather mock such dependency. Quick guide through adding
|
||||
mocks is provided later in this doc.
|
||||
|
||||
## Writing new tests
|
||||
In coreboot, [Cmocka](https://cmocka.org/) is used as unit test framework. The
|
||||
project has exhaustive [API documentation](https://api.cmocka.org/). Let's see
|
||||
how we may incorporate it when writing tests.
|
||||
In coreboot, [Cmocka](https://cmocka.org/) is used as unit test
|
||||
framework. The project has exhaustive [API
|
||||
documentation](https://api.cmocka.org/). Let's see how we may
|
||||
incorporate it when writing tests.
|
||||
|
||||
### Assertions
|
||||
Testing the UUT consists of calling the functions in the UUT and comparing the
|
||||
returned values to the expected values. Cmocka implements
|
||||
[a set of assert macros](https://api.cmocka.org/group__cmocka__asserts.html) to
|
||||
compare a value with an expected value. If the two values do not match, the test
|
||||
Testing the UUT consists of calling the functions in the UUT and
|
||||
comparing the returned values to the expected values. Cmocka implements
|
||||
[a set of assert
|
||||
macros](https://api.cmocka.org/group__cmocka__asserts.html) to compare a
|
||||
value with an expected value. If the two values do not match, the test
|
||||
fails with an error message.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
In our example, the simplest test is to call UUT for reading our fake devices
|
||||
registers and do all calculation in the test harness itself. At the end, let's
|
||||
compare integers with `assert_int_equal`.
|
||||
In our example, the simplest test is to call UUT for reading our fake
|
||||
devices registers and do all calculation in the test harness itself.
|
||||
At the end, let's compare integers with `assert_int_equal`.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
@@ -191,24 +201,25 @@ fails with an error message.
|
||||
### Mocks
|
||||
|
||||
#### Overview
|
||||
Many coreboot modules are low level software that touch hardware directly.
|
||||
Because of this, one of the most important and challenging part of
|
||||
writing tests is to design and implement mocks. A mock is a software component
|
||||
which implements the API of another component so that the test can verify that
|
||||
certain functions are called (or not called), verify the parameters passed to
|
||||
those functions, and specify the return values from those functions. Mocks are
|
||||
especially useful when the API to be implemented is one that accesses hardware
|
||||
components.
|
||||
Many coreboot modules are low level software that touch hardware
|
||||
directly. Because of this, one of the most important and challenging
|
||||
part of writing tests is to design and implement mocks. A mock is a
|
||||
software component which implements the API of another component so that
|
||||
the test can verify that certain functions are called (or not called),
|
||||
verify the parameters passed to those functions, and specify the return
|
||||
values from those functions. Mocks are especially useful when the API to
|
||||
be implemented is one that accesses hardware components.
|
||||
|
||||
When writing a mock, the developer implements the same API as the module being
|
||||
mocked. Such a mock may, for example, register a set of driver methods. Behind
|
||||
this API, there is usually a simulation of real hardware.
|
||||
When writing a mock, the developer implements the same API as the module
|
||||
being mocked. Such a mock may, for example, register a set of driver
|
||||
methods. Behind this API, there is usually a simulation of real
|
||||
hardware.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
For purpose of our i2c test, we may introduce two i2c devices with set of
|
||||
registers, which simply are structs in memory.
|
||||
For purpose of our i2c test, we may introduce two i2c devices with
|
||||
set of registers, which simply are structs in memory.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
@@ -266,16 +277,17 @@ this API, there is usually a simulation of real hardware.
|
||||
};
|
||||
```
|
||||
|
||||
Cmocka uses a feature that gcc provides for breaking dependencies at the link
|
||||
time. It is possible to override implementation of some function, with the
|
||||
method from test harness. This allows test harness to take control of execution
|
||||
from binary (during the execution of test), and stimulate UUT as required
|
||||
without changing the source code.
|
||||
Cmocka uses a feature that gcc provides for breaking dependencies at the
|
||||
link time. It is possible to override implementation of some function,
|
||||
with the method from test harness. This allows test harness to take
|
||||
control of execution from binary (during the execution of test), and
|
||||
stimulate UUT as required without changing the source code.
|
||||
|
||||
coreboot unit test infrastructure supports overriding of functions at link time.
|
||||
This is as simple as adding a `name_of_function` to be mocked into
|
||||
<test_name>-mocks variable in Makefile.inc. The result is that the test's
|
||||
implementation of that function is called instead of coreboot's.
|
||||
coreboot unit test infrastructure supports overriding of functions at
|
||||
link time. This is as simple as adding a `name_of_function` to be
|
||||
mocked into <test_name>-mocks variable in Makefile.inc. The result is
|
||||
that the test's implementation of that function is called instead of
|
||||
coreboot's.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
@@ -284,44 +296,45 @@ implementation of that function is called instead of coreboot's.
|
||||
|
||||
i2c-test-mocks += platform_i2c_transfer
|
||||
|
||||
Now, dev can write own implementation of `platform_i2c_transfer`. This
|
||||
implementation instead of accessing real i2c bus, will write/read from
|
||||
fake structs.
|
||||
Now, dev can write own implementation of `platform_i2c_transfer`.
|
||||
This implementation instead of accessing real i2c bus, will
|
||||
write/read from fake structs.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int platform_i2c_transfer(unsigned int bus, struct i2c_msg *segments,
|
||||
int count)
|
||||
int platform_i2c_transfer(unsigned int bus, struct i2c_msg
|
||||
*segments, int count)
|
||||
{
|
||||
}
|
||||
```
|
||||
|
||||
#### Checking mock's arguments
|
||||
A test can verify the parameters provided by the UUT to the mock function. The
|
||||
developer may also verify that number of calls to mock is correct and the order
|
||||
of calls to particular mocks is as expected (See
|
||||
[this](https://api.cmocka.org/group__cmocka__call__order.html)). The Cmocka
|
||||
macros for checking parameters are described
|
||||
[here](https://api.cmocka.org/group__cmocka__param.html). In general, in mock
|
||||
function, one makes a call to `check_expected(<param_name>)` and in the
|
||||
corresponding test function, `expect*()` macro, with description which parameter
|
||||
in which mock should have particular value, or be inside a described range.
|
||||
A test can verify the parameters provided by the UUT to the mock
|
||||
function. The developer may also verify that number of calls to mock is
|
||||
correct and the order of calls to particular mocks is as expected (See
|
||||
[this](https://api.cmocka.org/group__cmocka__call__order.html)). The
|
||||
Cmocka macros for checking parameters are described
|
||||
[here](https://api.cmocka.org/group__cmocka__param.html). In general, in
|
||||
mock function, one makes a call to `check_expected(<param_name>)` and in
|
||||
the corresponding test function, `expect*()` macro, with description
|
||||
which parameter in which mock should have particular value, or be inside
|
||||
a described range.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
In our example, we may want to check that `platform_i2c_transfer` is fed with
|
||||
number of segments bigger than 0, each segment has flags which are in
|
||||
supported range and each segment has buf which is non-NULL. We are expecting
|
||||
such values for _every_ call, thus the last parameter in `expect*` macros is
|
||||
-1.
|
||||
In our example, we may want to check that `platform_i2c_transfer` is
|
||||
fed with number of segments bigger than 0, each segment has flags
|
||||
which are in supported range and each segment has buf which is
|
||||
non-NULL. We are expecting such values for _every_ call, thus the
|
||||
last parameter in `expect*` macros is -1.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
static void mock_expect_params_platform_i2c_transfer(void)
|
||||
{
|
||||
unsigned long int expected_flags[] = {0, I2C_M_RD, I2C_M_TEN,
|
||||
I2C_M_RECV_LEN, I2C_M_NOSTART};
|
||||
unsigned long int expected_flags[] = {0, I2C_M_RD,
|
||||
I2C_M_TEN, I2C_M_RECV_LEN, I2C_M_NOSTART};
|
||||
|
||||
/* Flags should always be only within supported range */
|
||||
expect_in_set_count(platform_i2c_transfer, segments->flags,
|
||||
@@ -330,8 +343,8 @@ in which mock should have particular value, or be inside a described range.
|
||||
expect_not_value_count(platform_i2c_transfer, segments->buf,
|
||||
NULL, -1);
|
||||
|
||||
expect_in_range_count(platform_i2c_transfer, count, 1, INT_MAX,
|
||||
-1);
|
||||
expect_in_range_count(platform_i2c_transfer, count, 1,
|
||||
INT_MAX, -1);
|
||||
}
|
||||
|
||||
And the checks below should be added to our mock
|
||||
@@ -347,11 +360,11 @@ in which mock should have particular value, or be inside a described range.
|
||||
```
|
||||
|
||||
#### Instrument mocks
|
||||
It is possible for the test function to instrument what the mock will return to
|
||||
the UUT. This can be done by using the `will_return*()` and `mock()` macros.
|
||||
These are described in
|
||||
[the Mock Object section](https://api.cmocka.org/group__cmocka__mock.html) of
|
||||
the Cmocka API documentation.
|
||||
It is possible for the test function to instrument what the mock will
|
||||
return to the UUT. This can be done by using the `will_return*()` and
|
||||
`mock()` macros. These are described in [the Mock Object
|
||||
section](https://api.cmocka.org/group__cmocka__mock.html) of the Cmocka
|
||||
API documentation.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: Example
|
||||
@@ -361,17 +374,18 @@ the Cmocka API documentation.
|
||||
```
|
||||
|
||||
### Test runner
|
||||
Finally, the developer needs to implement the test `main()` function. All tests
|
||||
should be registered there and cmocka test runner invoked. All methods for
|
||||
invoking Cmocka test are described
|
||||
Finally, the developer needs to implement the test `main()` function.
|
||||
All tests should be registered there and cmocka test runner invoked. All
|
||||
methods for invoking Cmocka test are described
|
||||
[here](https://api.cmocka.org/group__cmocka__exec.html).
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
We don't need any extra setup and teardown functions for i2c-test, so let's
|
||||
simply register test for `i2c_read_field` and return from main value which is
|
||||
output of Cmocka's runner (it returns number of tests that failed).
|
||||
We don't need any extra setup and teardown functions for i2c-test, so
|
||||
let's simply register test for `i2c_read_field` and return from main
|
||||
value which is output of Cmocka's runner (it returns number of tests
|
||||
that failed).
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
|
Reference in New Issue
Block a user