Hello everyone, how's it going?
In this post I'd like to explain and teach how to enable debugging of system components, Click apps as well as Snap packages. This is intended for developers choosing to contribute to the Operating System's development, wanting to fix a bug in their Click apps, or release Ubuntu Touch apps on the Snap Store for both Ubuntu Touch and all Snap-supporting Linux distributions.
Prerequisites
The Snap Store offers a static version of gdb which we're going to make use of for this setup:
sudo snap install gdb-static
Make sure to add this snippet to your ~/.bashrc file:
export PATH="/snap/gdb-static/current/usr/bin:$PATH"
After logging into a shell you are now able to run the Snap-provided gdb and gdbserver commands directly, but they don't yet work when using sudo with short-hand commands like sudo gdb.
Running gdb & strace with sudo
Since these versions of debugging tools don't sit in a PATH that is allowed for sudo-invoked commands, we will have to teach sudo to set them in its default PATH without breaking the Operating System's expectations of the read-only rootfs. So to make this work, we will have to create an overlay for /etc/sudoers.d which configures sudo to keep our tools paths intact.
First let's create an overlay containing our sudoers addition:
sudo mkdir -p /userdata/custom/overlays/etc/sudoers.d
cat - | sudo tee /userdata/custom/overlays/etc/sudoers.d/gdb <<EOF
Defaults secure_path="/snap/gdb-static/current/usr/bin:/snap/strace-static/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
EOF
sudo chmod 440 /userdata/custom/overlays/etc/sudoers.d/gdb
sudo chmod 500 /userdata/custom/overlays/etc/sudoers.d
This effectively configures sudo to keep the paths to gdb, gdbserver and strace (described later) when executing i.e sudo gdb or when snap run is launched with the --gdbserver argument.
Next create a systemd mount unit to effectively apply this overlay immediately and enable on boot:
cat - | sudo tee /etc/systemd/system/etc-sudoers.d.mount <<EOF
[Unit]
Before=local-fs.target
[Mount]
What=/userdata/custom/overlays/etc/sudoers.d
Where=/etc/sudoers.d
Type=overlay
Options=ro,relatime,lowerdir=/userdata/custom/overlays/etc/sudoers.d:/etc/sudoers.d
[Install]
WantedBy=local-fs.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now etc-sudoers.d.mount
This overlay is not modifying the immutable file system and is secure against Operating System updates changing the underlying files and defaults other than what we adapted.
Debugging your app
You're now able to attach to your desired application's process, simply using sudo gdb -p <PID-of-the-process>, whether it being a system process or Click app.
You can also safely debug your Snaps using built-in commands. Let's take the wasted Snap as an example:
snap run --gdbserver wasted
This will prompt you to 1) enter your sudo password and 2) to execute the outlined gdb command in a separate terminal:
phablet@fairphone5:~$ snap run --gdbserver wasted
Welcome to "snap run --gdbserver".
You are right before your application is run.
Please open a different terminal and run:
gdb -ex="target remote :42957" -ex=continue -ex="signal SIGCONT"
(gdb) continue
or use your favorite gdb frontend and connect to :42957
Running the mentioned command and typing cont in the resulting gdb session will launch the debugging process. You are also free to connect to the server's TCP port it mentions from your development PC, allowing you to comfortably and remotely debug the app.
Tip: If your app receives SIGSTOP signals frequently during start-up, filter those signals out of process management:
(gdb) handle SIGSTOP nostop
syscall tracing
The Snap Store also has strace-static in store for debugging a process' use of syscalls at runtime. Since we've already set up the strace-static PATH in the /etc/sudoers.d overlay we can proceed with the installation:
sudo snap install strace-static
If you wish to use this version of strace over the system default one, add this to your ~/.bashrc:
export PATH="/snap/strace-static/current/bin:$PATH"
You can now simply run a command with strace prefixed, or attach to all running process' threads and children using sudo strace -p <PID-of-the-process> -ff. All the regularly expected strace command arguments apply.
On the Snap side, with a simple snap run --strace wasted we're able to capture each syscall the Snap makes to the kernel. snap run --strace wasted 2>strace.log will save the captured trace to strace.log for further inspection.
How does this work?
Unlike an apt-installed version of gdb, this one is an all-it-can-support static build of gdb into one file, same as strace-static for strace. This means those can be executed on any Linux OS, regardless of dependencies provided by your host's OS. But since they aren't dynamic binaries instead they cannot be extended by various dependencies through packaging means, especially because in this case the required sources currently expect to be compiled & linked dynamically.
As a result of all this gdb-static lacks debuginfod support for debug symbol retrieval because libdebuginfod upstream still requires changes to build statically.
Final words
You did it! You have successfully set up your Ubuntu Touch for debugging purposes without sacrificing on the system's immutability, OTA installation guarantee, or in ease-of-use.
We just had to override the system-default PATH for regular sudo commands in a way that would survive any OTA coming in, and without touching our beloved immutable file system.
Congratulations, you now know how to debug on Ubuntu Touch.
