OTA Connect Developer Guide

Running aktualizr with fault injection

To test the system in adverse conditions, it can be useful to make aktualizr fail in a controlled fashion.

libfiu provides a framework to do that, and aktualizr supports a number of controllable fail points.

Setup

libfiu needs to be installed on the target machine. For local compilation on a Debian-derived system, the distribution package can be used:

apt install fiu-utils libfiu-dev

Fault injection must then be enabled at CMake configure time with the -DFAULT_INJECTION=on option (refer to general building instructions for more details).

fiu-run and fiu-ctrl can now be used on the newly compiled aktualizr binary to inject faults (refer to corresponding man pages).

For example, when using the fake package manager:

fiu-run -c 'enable name=fake_package_install' aktualizr -c . once

Our wrapper script in ./scripts/fiu can be used in place, with the added feature of passing strings in failinfo parameters.

Usage is as follow:

./scripts/fiu run -c 'enable name=fake_package_install,failinfo=reason' -- aktualizr -c . once

List of fail points

Please try to keep this list up-to-date when inserting/removing fail points.

  • fake_package_download: force the fake package manager download to fail

  • fake_package_install: force the fake package manager installation to fail

  • fake_install_finalization_failure: force the fake package manager installation finalization to fail

  • secondary_putmetadata: force virtual Secondary metadata verification to fail

  • secondary_putroot: force virtual Secondary Root rotation to fail

  • secondary_sendFirmware_xxx (xxx is a virtual Secondary ECU ID): force a virtual Secondary firmware send to fail

  • secondary_install_xxx (xxx is a virtual Secondary ECU ID): force a virtual Secondary installation to fail

Use in unit tests

It is encouraged to use fail points to help unit testing sad paths. Tests that require fault injection should only be run if the FIU_ENABLE macro is defined.

Example with a docker container

The aktualizr application docker image is compiled with fault injection support. It can be used as a quick way to simulate intermittent installation failures.

First build the image:

./docker/docker-build.sh

It will tag an image advancedtelematic/aktualizr-app:latest which includes the common aktualizr tools, as well as the fiu tools (fiu-run, fiu-ctrl…​)

Then, prepare an environment for a simulated device, following the article Simulate a device without building a disk image.

aktualizr will be run through docker instead of the local system, so all commands will start with:

docker run -u $(id -u):$(id -g) -w $PWD -v $PWD:$PWD advancedtelematic/aktualizr-app

It runs the docker image inside a container with the same permissions as the local user, with a volume mounted in the current local directory.

Then, let’s launch aktualizr with fiu-run:

docker run --name aktualizr-fiu --rm -u $(id -u):$(id -g) -w $PWD -v $PWD:$PWD advancedtelematic/aktualizr-app fiu run -- aktualizr -c .

You can try to install a package now, which will succeed. To make all subsequent installations fail with the reason "TEST_FAILURE", use:

docker exec aktualizr-fiu fiu ctrl -c 'enable name=fake_package_install,failinfo=TEST_FAILURE' 1

To make installations succeed again:

docker exec aktualizr-fiu fiu ctrl -c 'disable name=fake_package_install' 1