OTA Connect Developer Guide

Aktualizr Debugging Tips

Running Aktualizr in development

The sota-local.toml configuration file sets the package manager to PackageManagerFake, which allows the installation process to be tested locally:

# From the build directory
mkdir sota-prov
src/aktualizr_primary/aktualizr --config ../config/sota-local.toml

For VS code, there is a launch.json provided in the .vscode directory. This assumes that the cmake build directory is build.

Creating a temporary OSTree environment

Try the scripts available in the tests/ostree-scripts directory.

Inspect stored info with aktualizr-info

The aktualizr-info tool can be used to dump information stored in the libaktualizr database. By default, it displays basic information such as storage type, device ID, primary ECU serial and hardware ID and provisioning status. Additional information can be requested with various command line parameters.

Valgrind and gdb

If the target application or test is running under valgrind, then gdb can still be connected to the process without stopping it. First run vgdb --port=2159 in a different shell on the same machine, then connect to it using target remote localhost:2159 in gdb

Dumping SSL encrypted traffic

First fetch and build sslkeylog from https://github.com/cajun-rat/sslkeylog

Next copy aktualizr.service to /etc/systemd/system/aktualizr.service. This will override the packages default config.

Now modify /etc/systemd/system/aktualizr.service and add the following lines:

Environment=SSLKEYLOGFILE=/var/sota/premaster.txt
Environment=LD_PRELOAD=/usr/lib/libsslkeylog.so

Reload the config and restart

# systemctl daemon-reload
# systemctl restart aktualizr

The symmetric SSL keys will be logged in /var/sota/premaster.txt

Now capture a packet dump with tcpdump

sudo tcpdump tcp port 443 -w upload.pcap

Fetch both of these down. In wireshark preferences set ssl.keylog_file to point to premaster.txt. If your https traffic not on port 443, then add the port to http.ssl.port. Now open upload.pcap.

Serve repo generated by aktualizr-repo

aktualizr can be tested against a dummy repository containing fake images.

First, generate a repository using aktualizr-repo tool:

aktualizr-repo generate <repo_dir>

Then, serve the generated directory using a web server such as the fake test server.

For more information about using aktualizr-repo, see aktualizr-repo.adoc.

Here is an example configuration for nginx:

server {
    listen 80;
    listen [::]:80;
    server_name localhost;

    location / {
        try_files  $request_uri $request_uri;
    }
    location /director/manifest {
        try_files  $request_uri $request_uri;
        dav_methods  PUT;
    }

    root repo_dir/repo;
}

Inject faults

Developing and debugging with an OpenEmbedded system

By default OpenEmbedded builds fixed versions of software from a VCS using bitbake recipes. When developing Aktualizr itself it is useful to have a quicker edit-compile-run cycle and access to a debugger. The following steps will use OpenEmbedded to create a cross-compilation environment, then build inside that.

  1. Add the following to local.conf:

    TOOLCHAIN_HOST_TASK_append = " nativesdk-cmake "
  2. Build the SDK:

    bitbake -c populate_sdk core-image-minimal
  3. That will create a self-extracting installer that can be copied to your development machine. Install it by executing this script (or a similarly-named one, depending on your environment):

    ./tmp/deploy/sdk/poky-sota-glibc-x86_64-core-image-minimal-core2-64-toolchain-2.2.2.sh
  4. Execute this script (or something similar, depending on where you installed it) to update the environment to point to the cross compilers:

    . /opt/poky-sota/2.2.2/environment-setup-core2-64-poky-linux

    You may want to verify that which cmake returns something like this:

    /opt/poky-sota/2.2.2/sysroots/x86_64-pokysdk-linux/usr/bin/cmake
  5. Create a cmake build directory for this cross-compile:

    mkdir build-cross
    cd build-cross
    cmake .. <options>
    make aktualizr

The compiled 'aktualizr' executable can be copied to the remote system and run.

Aktualizr can be debugged remotely by exposing a port from the VM to development machine (the --gdb option to the run-qemu-ota script in meta-updater does this), then:

On the target:
gdbserver 0.0.0.0:2159 ./aktualizr --config /usr/lib/sota/sota.toml --loglevel 0
On the host:
$ gdb aktualizr
(gdb) target remote localhost:2159

In CLion the remote debugger is configured as follows:

CLion GDB configuration

It is also possible to run it inside valgrind:

On the target:
valgrind --vgdb=yes --vgdb-error=0 ./aktualizr --config /usr/lib/sota/sota.toml
vgdb --port=2159

Then connect the debugger as usual.

Bitbaking with debug symbols

For every binary you want to debug (executables and shared libraries alike) you need to add these two lines in the recipe:

INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
INHIBIT_PACKAGE_STRIP = "1"

You also need to make it build with debug symbols, which is recipe-dependent. For aktualizr it means specifying

-DCMAKE_BUILD_TYPE=Debug

instead of Release. However, this method does not install the sources on the device, so it helps to open the source file on your host machine.