Thursday, February 18, 2016

Trace QEMU with LTTng UST under libvirt (OpenStack)

In a previous blog post, I showed how to configure QEMU for tracing using LTTng UST. I showed that when launching manually a virtual machine, LTTng should list the existing tracepoints within QEMU.
It has been brought to my attention that tracepoints aren't "visible" when virtual machines are started from virt-manager, OpenStack, or any way other than manually launching the QEMU binary.
The symptoms are as follows:
  • Start QEMU manually, for instance: $> qemu-system-x86_64
  • List UST tracepoints: $> lttng list -u
  • LTTng will list all available tracepoints
  • Stop QEMU and start a VM from virt-manager or OpenStack
  • List UST tracepoints: $> lttng list -u
  • LTTng doesn't list any tracepoint
The first thing to check is that it is the correct QEMU binary that is launched. You might have multiple versions of QEMU installed (for instance, one from your distro's package, and one that you configured and built yourself). To verify that, find the PID of the QEMU process, and look at the file /proc/$PID/cmdline. This will show you, among other things, the location of the QEMU binary used to launch that VM. Typically, if you built and installed QEMU yourself using make and make install, the binary should be under /usr/local/bin.

Assuming that is taken care of, tracepoints will still not be shown from LTTng UST when VMs are started from other applications. The reason is that virtual machines are actually launched by libvirt, a middleware between VM management applications (virt-manager, OpenStack, etc) and the VMs themselves. The problem with that is that the libvirt daemon launches the VMs under the user account libvirt-qemu.
By default, tracing with LTTng is only allowed for root or for users who belong to the tracing group. Since the libvirt-qemu user doesn't belong to that group, it doesn't have the privileges to control tracing using LTTng. The workaround is to add the user libvirt-qemu to the tracing group, as such:

  
    $> sudo usermod -a -G tracing libvirt-qemu
  

Restart the libvirt daemon, launch your VMs and tracing should now work from LTTng UST.

6 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. After running:

    adduser libvirt-qemu tracing
    reboot

    And starting VMs with Libvirt, or even manually by calling qemu-system-x86_64 directly, lttng doesn't work anymore...

    I'm seeing now:

    ---
    lttng list -u
    Error: Unable to list UST events: No session daemon is available
    Error: Command error
    ---

    Any idea? =P

    ReplyDelete
    Replies
    1. Mmm... Never mind... After running (Ubuntu 16.04):

      apt install lttng-tools lttng-modules-dkms --reinstall

      It is working again, some service not enabled on boot? Maybe... let me see if I can trace vms started by libvirt...

      Delete
    2. Still can't trace VMs launched by libvirt/virt-manager... ;-(

      Delete
    3. What is your current state? What happens if you do "lttng list -u" while the VM is running? You should at least make sure that the lttng-sessiond daemon is running, as such:

      $> pgrep -l lttng-sessiond

      which should return the PID of the daemon. If that's not the case, you can launch it by doing:

      $> sudo lttng-sessiond -d

      Delete
  3. This comment has been removed by the author.

    ReplyDelete

Registering a probe to a kernel module using Systemtap

I was trying to register a probe on a function of a kernel module of mine using Systemtap. The .stp file was fairly simple: $> cat mymod...