Trace QEMU with LTTng UST under libvirt (OpenStack)

In a previous blog post, I showed how to configure QEMU for tracing using LTTng UST. I showed that when launching manually a virtual machine, LTTng should list the existing tracepoints within QEMU.
It has been brought to my attention that tracepoints aren't "visible" when virtual machines are started from virt-manager, OpenStack, or any way other than manually launching the QEMU binary.
The symptoms are as follows:

• Start QEMU manually, for instance: $> qemu-system-x86_64
• List UST tracepoints: $> lttng list -u
• LTTng will list all available tracepoints
• Stop QEMU and start a VM from virt-manager or OpenStack
• List UST tracepoints: $> lttng list -u
• LTTng doesn't list any tracepoint

The first thing to check is that it is the correct QEMU binary that is launched. You might have multiple versions of QEMU installed (for instance, one from your distro's package, and one that you configured and built yourself). To verify that, find the PID of the QEMU process, and look at the file /proc/$PID/cmdline. This will show you, among other things, the location of the QEMU binary used to launch that VM. Typically, if you built and installed QEMU yourself using make and make install, the binary should be under /usr/local/bin.

Assuming that is taken care of, tracepoints will still not be shown from LTTng UST when VMs are started from other applications. The reason is that virtual machines are actually launched by libvirt, a middleware between VM management applications (virt-manager, OpenStack, etc) and the VMs themselves. The problem with that is that the libvirt daemon launches the VMs under the user account libvirt-qemu.
By default, tracing with LTTng is only allowed for root or for users who belong to the tracing group. Since the libvirt-qemu user doesn't belong to that group, it doesn't have the privileges to control tracing using LTTng. The workaround is to add the user libvirt-qemu to the tracing group, as such:

  
    $> sudo usermod -a -G tracing libvirt-qemu
  

Restart the libvirt daemon, launch your VMs and tracing should now work from LTTng UST.

Follow @mogeb88

How to trace QEMU using LTTng (UST)

QEMU source code is instrumented with many tracepoints and supports multiple userspace tracers. Depending on which backend QEMU was configured for, you can use dtrace, ftrace, etc. as a tracer. A few months back, I submitted a patch to QEMU to support LTTng's new ABI after its 2.0 version. However, most Linux distributions ship QEMU configured to use other tracers as a backend. In this blog post, I will show how one can use LTTng as a backend to trace QEMU in userspace.
LTTng is a low-latency kernel and userspace (also known as UST - UserSpace Tracer) tracer. You can read more about it on www.lttng.org, or find basic documentation and many articles about its internals. LTTng records traces in the Common Trace Format (CTF), which can later be viewed using Babeltrace or Eclipse's Tracing and Monitoring Framework (TMF). To use LTTng UST as a backend for QEMU, you first need to get the source code of QEMU and configure it accordingly. You can either get the sources by cloning QEMU's git repo, or by getting the sources from your distribution's repo:

$> git clone git://git.qemu.org/qemu.git

or

$> apt-get source qemu

if you are using a Debian-based distribution. Once QEMU's source code is downloaded you can view its configuration options:

$> ./configure --help

If you search for "enable-trace-backend" within these options, you will find a list of tracers which can be used as a backend for QEMU, including "ust". We can then configure QEMU for this particular tracer:

$> ./configure --enable-trace-backend=ust

You might need to install libpixman-1-dev and libfdt-dev before the configuration succeeds. Once QEMU is configured with UST as its tracing backend, simply compile:

$> make -j8

And run QEMU to test it (assuming you have LTTng installed and set):

$> ./x86_64-softmmu/qemu-system-x86_64 &

We can now run LTTng UST to make sure it can "see" QEMU's tracepoints:

$> lttng list -u

The result should be a long list of tracepoints. You can now use LTTng as you usually would. Good tracing!

Follow @mogeb88