Running aktualizr with fault injection
To test the system in adverse conditions, it can be useful to make aktualizr fail in a controlled fashion.
libfiu provides a framework to do that, and aktualizr supports a number of controllable fail points.
Setup
libfiu needs to be installed on the target machine. For local compilation on a Debian-derived system, the distribution package can be used:
apt install fiu-utils libfiu-dev
Fault injection must then be enabled at CMake configure time with the -DFAULT_INJECTION=on
option (refer to general building instructions for more details).
fiu-run
and fiu-ctrl
can now be used on the newly compiled aktualizr binary to inject faults (refer to corresponding man pages).
For example, when using the fake package manager:
fiu-run -c 'enable name=fake_package_install' aktualizr -c . once
Our wrapper script in ./scripts/fiu
can be used in place, with the added feature of passing strings in failinfo parameters.
Usage is as follow:
./scripts/fiu run -c 'enable name=fake_package_install,failinfo=reason' -- aktualizr -c . once
List of fail points
Please try to keep this list up-to-date when inserting/removing fail points.
-
fake_package_download
: force the fake package manager download to fail, optionally with a failure code supplied viafailinfo
-
fake_package_install
: force the fake package manager installation to fail, optionally with a failure code supplied viafailinfo
-
secondary_install_xxx
(xxx is a virtual secondary ecu id): force a virtual secondary installation to fail, optionally with a failure code supplied viafailinfo
-
fake_install_finalization_failure
: force the fake package manager installation finalization to fail
Use in unit tests
It is encouraged to use fail points to help unit testing sad paths. Tests that require fault injection should only be run if the FIU_ENABLE
macro is defined.
Example with a docker container
The aktualizr application docker image is compiled with fault injection support. It can be used as a quick way to simulate intermittent installation failures.
First build the image:
./docker/docker-build.sh
It will tag an image advancedtelematic/aktualizr-app:latest
which includes the common aktualizr tools, as well as the fiu tools (fiu-run
, fiu-ctrl
…)
Then, prepare an environment for a simulated device, following the article Simulate a device without building a disk image.
aktualizr
will be run through docker instead of the local system, so all commands will start with:
docker run -u $(id -u):$(id -g) -w $PWD -v $PWD:$PWD advancedtelematic/aktualizr-app
It runs the docker image inside a container with the same permissions as the local user, with a volume mounted in the current local directory.
Then, let’s launch aktualizr with fiu-run
:
docker run --name aktualizr-fiu --rm -u $(id -u):$(id -g) -w $PWD -v $PWD:$PWD advancedtelematic/aktualizr-app fiu run -- aktualizr -c .
You can try to install a package now, which will succeed. To make all subsequent installations fail with the reason "TEST_FAILURE", use:
docker exec aktualizr-fiu fiu ctrl -c 'enable name=fake_package_install,failinfo=TEST_FAILURE' 1
To make installations succeed again:
docker exec aktualizr-fiu fiu ctrl -c 'disable name=fake_package_install' 1