Registering a probe to a kernel module using Systemtap

I was trying to register a probe on a function of a kernel module of mine using Systemtap. The .stp file was fairly simple:

$> cat mymod.stp
probe module("mymodule").function("myfunction") {
  now_time = gettimeofday_ns()
  print(now_time, "\n")
}

When I was trying to register the probe, Systemtap kept failing:

$> sudo stap mymod.stp
semantic error: while resolving probe point: identifier 'module' at mymod.stp:1:7
        source: probe module("mymodule").function("myfunction") {
                      ^

semantic error: no match

Pass 2: analysis failed.  [man error::pass2]

Yet my module was properly loaded:

$> lsmod | grep mymodule
mymodule               40960  0

And the symbol was valid:

$> grep myfunction /proc/kallsyms
0000000000000000 t myfunction    [mymodule]

Turns out you have to actually install the probe under /lib/modules/KERNEL_VERSION for Systemtap to be able to register the probe on it. The final fix:

$> sudo rmmod mymodule
$> sudo install ~/build/mymodule.ko /lib/modules/4.5.0/extra/
$> sudo modprobe mymodule

Be sure to replace 4.5.0 by your current kernel's version:

$> sudo install ~/build/mymodule.ko /lib/modules/`uname -r`/extra/

Follow @mogeb88

Trace QEMU with LTTng UST under libvirt (OpenStack)

In a previous blog post, I showed how to configure QEMU for tracing using LTTng UST. I showed that when launching manually a virtual machine, LTTng should list the existing tracepoints within QEMU.
It has been brought to my attention that tracepoints aren't "visible" when virtual machines are started from virt-manager, OpenStack, or any way other than manually launching the QEMU binary.
The symptoms are as follows:

• Start QEMU manually, for instance: $> qemu-system-x86_64
• List UST tracepoints: $> lttng list -u
• LTTng will list all available tracepoints
• Stop QEMU and start a VM from virt-manager or OpenStack
• List UST tracepoints: $> lttng list -u
• LTTng doesn't list any tracepoint

The first thing to check is that it is the correct QEMU binary that is launched. You might have multiple versions of QEMU installed (for instance, one from your distro's package, and one that you configured and built yourself). To verify that, find the PID of the QEMU process, and look at the file /proc/$PID/cmdline. This will show you, among other things, the location of the QEMU binary used to launch that VM. Typically, if you built and installed QEMU yourself using make and make install, the binary should be under /usr/local/bin.

Assuming that is taken care of, tracepoints will still not be shown from LTTng UST when VMs are started from other applications. The reason is that virtual machines are actually launched by libvirt, a middleware between VM management applications (virt-manager, OpenStack, etc) and the VMs themselves. The problem with that is that the libvirt daemon launches the VMs under the user account libvirt-qemu.
By default, tracing with LTTng is only allowed for root or for users who belong to the tracing group. Since the libvirt-qemu user doesn't belong to that group, it doesn't have the privileges to control tracing using LTTng. The workaround is to add the user libvirt-qemu to the tracing group, as such:

  
    $> sudo usermod -a -G tracing libvirt-qemu
  

Restart the libvirt daemon, launch your VMs and tracing should now work from LTTng UST.

Follow @mogeb88