Aktualizr Debugging Tips
Running Aktualizr in development
The sota-local.toml
configuration file sets the package manager to PackageManagerFake
, which allows the installation process to be tested locally:
# From the build directory mkdir sota-prov src/aktualizr_primary/aktualizr --config ../config/sota-local.toml
For VS code, there is a launch.json
provided in the .vscode
directory. This assumes that the cmake build directory is build
.
Creating a temporary OSTree environment
Try the scripts available in the tests/ostree-scripts directory.
Inspect stored info with aktualizr-info
The aktualizr-info tool can be used to dump information stored in the libaktualizr database. By default, it displays basic information such as storage type, device ID, primary ECU serial and hardware ID and provisioning status. Additional information can be requested with various command line parameters.
Valgrind and gdb
If the target application or test is running under valgrind, then gdb can still be connected to the process without stopping it. First run vgdb --port=2159
in a different shell on the same machine, then connect to it using target remote localhost:2159
in gdb
Dumping SSL encrypted traffic
First fetch and build sslkeylog from https://github.com/cajun-rat/sslkeylog
Next copy aktualizr.service to /etc/systemd/system/aktualizr.service. This will override the packages default config.
Now modify /etc/systemd/system/aktualizr.service
and add the following lines:
Environment=SSLKEYLOGFILE=/var/sota/premaster.txt
Environment=LD_PRELOAD=/usr/lib/libsslkeylog.so
Reload the config and restart
# systemctl daemon-reload
# systemctl restart aktualizr
The symmetric SSL keys will be logged in /var/sota/premaster.txt
Now capture a packet dump with tcpdump
sudo tcpdump tcp port 443 -w upload.pcap
Fetch both of these down. In wireshark preferences set ssl.keylog_file to point to premaster.txt. If your https traffic not on port 443, then add the port to http.ssl.port. Now open upload.pcap.
Serve repo generated by uptane-generator
aktualizr can be tested against a dummy repository containing fake images.
First, generate a repository using uptane-generator tool:
uptane-generator generate <repo_dir>
Then, serve the generated directory using a web server such as the fake test server.
For more information about using uptane-generator, see uptane-generator.adoc.
Here is an example configuration for nginx:
server {
listen 80;
listen [::]:80;
server_name localhost;
location / {
try_files $request_uri $request_uri;
}
location /director/manifest {
try_files $request_uri $request_uri;
dav_methods PUT;
}
root repo_dir/repo;
}
Developing and debugging with an OpenEmbedded system
By default OpenEmbedded builds fixed versions of software from a VCS using bitbake recipes. When developing Aktualizr itself it is useful to have a quicker edit-compile-run cycle and access to a debugger. The following steps will use OpenEmbedded to create a cross-compilation environment, then build inside that.
-
Add the following to local.conf:
TOOLCHAIN_HOST_TASK_append = " nativesdk-cmake "
-
Build the SDK:
bitbake -c populate_sdk core-image-minimal
-
That will create a self-extracting installer that can be copied to your development machine. Install it by executing this script (or a similarly-named one, depending on your environment):
./tmp/deploy/sdk/poky-sota-glibc-x86_64-core-image-minimal-core2-64-toolchain-2.2.2.sh
-
Execute this script (or something similar, depending on where you installed it) to update the environment to point to the cross compilers:
. /opt/poky-sota/2.2.2/environment-setup-core2-64-poky-linux
You may want to verify that
which cmake
returns something like this:/opt/poky-sota/2.2.2/sysroots/x86_64-pokysdk-linux/usr/bin/cmake
-
Create a cmake build directory for this cross-compile:
mkdir build-cross cd build-cross cmake .. <options> make aktualizr
The compiled 'aktualizr' executable can be copied to the remote system and run.
Aktualizr can be debugged remotely by exposing a port from the VM to development machine (the --gdb option to the run-qemu-ota script in meta-updater does this), then:
gdbserver 0.0.0.0:2159 ./aktualizr --config /usr/lib/sota/sota.toml --loglevel 0
$ gdb aktualizr (gdb) target remote localhost:2159
In CLion the remote debugger is configured as follows:
It is also possible to run it inside valgrind:
valgrind --vgdb=yes --vgdb-error=0 ./aktualizr --config /usr/lib/sota/sota.toml vgdb --port=2159
Then connect the debugger as usual.
Bitbaking with debug symbols
For every binary you want to debug (executables and shared libraries alike) you need to add these two lines in the recipe:
INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
INHIBIT_PACKAGE_STRIP = "1"
You also need to make it build with debug symbols, which is recipe-dependent. For aktualizr it means specifying
-DCMAKE_BUILD_TYPE=Debug
instead of Release. However, this method does not install the sources on the device, so it helps to open the source file on your host machine.