Sunday, August 31, 2014

How to trace QEMU using LTTng (UST)

QEMU source code is instrumented with many tracepoints and supports multiple userspace tracers. Depending on which backend QEMU was configured for, you can use dtrace, ftrace, etc. as a tracer. A few months back, I submitted a patch to QEMU to support LTTng's new ABI after its 2.0 version. However, most Linux distributions ship QEMU configured to use other tracers as a backend. In this blog post, I will show how one can use LTTng as a backend to trace QEMU in userspace.
LTTng is a low-latency kernel and userspace (also known as UST - UserSpace Tracer) tracer. You can read more about it on www.lttng.org, or find basic documentation and many articles about its internals. LTTng records traces in the Common Trace Format (CTF), which can later be viewed using Babeltrace or Eclipse's Tracing and Monitoring Framework (TMF). To use LTTng UST as a backend for QEMU, you first need to get the source code of QEMU and configure it accordingly. You can either get the sources by cloning QEMU's git repo, or by getting the sources from your distribution's repo:
$> git clone git://git.qemu.org/qemu.git
or
$> apt-get source qemu
if you are using a Debian-based distribution. Once QEMU's source code is downloaded you can view its configuration options:
$> ./configure --help
If you search for "enable-trace-backend" within these options, you will find a list of tracers which can be used as a backend for QEMU, including "ust". We can then configure QEMU for this particular tracer:
$> ./configure --enable-trace-backend=ust
You might need to install libpixman-1-dev and libfdt-dev before the configuration succeeds. Once QEMU is configured with UST as its tracing backend, simply compile:
$> make -j8
And run QEMU to test it (assuming you have LTTng installed and set):
$> ./x86_64-softmmu/qemu-system-x86_64 &
We can now run LTTng UST to make sure it can "see" QEMU's tracepoints:
$> lttng list -u
The result should be a long list of tracepoints. You can now use LTTng as you usually would. Good tracing!

4 comments:

  1. Does lttng support virtio-trace for lower overhead?

    ReplyDelete
    Replies
    1. No, unfortunately it doesn't. But I agree that it would be a huge improvement for Virtual Machines tracing. :)

      Delete
  2. Does lttng allow instruction tracing, including memory accesses (i.e. memory access location and value)?

    ReplyDelete
    Replies
    1. Unfortunately it doesn't.. However, LTTng can give you PMU counters values, such as cache misses (including L1 and LLC) and the number of instructions. I think you can use perf to trace memory accesses.

      Delete

Registering a probe to a kernel module using Systemtap

I was trying to register a probe on a function of a kernel module of mine using Systemtap. The .stp file was fairly simple: $> cat mymod...