The magic-trace direct backend and ideas for the future

The direct_backend subdirectory of magic-trace contains an alternative backend which directly uses perf_event_open and the libipt library to capture the Processor Trace data and decode it. This is faster for decoding than going via the perf command, and gives more control, but had more gotchas and needed more work than initially expected.

It works, but currently it doesn't support decoding usage of the Linux vDSO, which means whenever a vDSO syscall like getting the current time happens, it causes a trace decoding error leading to a short gap in the trace. It's also missing many other features perf has like multi-threaded recording and capturing multiple snapshots.

It remains here because it represents a substantial investment in figuring out how to directly use perf_event_open with Intel Processor Trace and libipt together, the only such example code I know of. Should anyone want to do something that requires deeper control over Processor Trace, they may need this code. That's also why it's open-sourced despite it not (yet) being set up to build outside the Jane Street tree, because it's potentially valuable reference code.

The demand for instruction-level information

The main reason we started down the path to this backend is that the basic perf backend of magic-trace doesn't properly follow the effect of OCaml exceptions on the stack. Doing so requires noticing the instructions that push, pop and raise from the OCaml exception handler stack, which aren't branches and so aren't listed in perf's branch output.

The initial purpose of the libipt backend, as well as increased decoding performance, was enabling instruction-by-instruction decoding so we could follow exceptions properly.

An alternative approach

During the process of implementing the direct backend, it became clear that the task was more difficult than we thought, and a picture of an alternative approach emerged, on that seemed like potentially less work than bringing the direct backend up to where perf is.

Here's a sketch about what the future of magic-trace could look like:

perf dlfilter

Newer versions of perf have a feature called perf-dlfilter that allows perf to load a shared library which gets callbacks for decoded events using a C API. This was specifically designed to allow consuming Intel Processor Trace data very quickly and without text parsing.

We could implement a small shared library in something that's easy to connect with a C interface, like C/Rust/Zig, that does the following:

• Filter out jumps that are within the same symbol, currently we need to process and discard all of these in OCaml, which is inefficient.
• Write out info about the relevant events in an efficient binary format that's easy to consume from OCaml, potentially even Fuchsia Trace Format.

This would achieve the goal of improving decoding performance with much less tricky C code and without reimplementing so many perf features.

perf --control

Currently we tell perf to take snapshots by sending it SIGUSR2. This requires some arbitrary wait timers and has some issues reliably capturing snapshots when a process shuts down. The direct backend's use of perf_event_open was going to solve this.

However newer versions of perf add a --control flag that allows using a FIFO with acks to toggle events and take snapshots, removing the need for signals and waits, and also a --snapshot=e option to guarantee a snapshot on end if there hasn't been another snapshot.

instruction-level decoding using basic blocks

In order to handle OCaml exceptions we need instruction-level decoding. This could've been done with libipt but an alternative way is to use a separate instruction decoding library/tool to decode the instructions between branch events provided by perf.

This would look something like having a cache of computed info for each previously encountered basic block, for example the exception handler pushes and pops present in it. When a new basic block (start and end address of execution with no intervening branches) is encountered, that range of the binary is decoded with a disassembler library and the necessary info computed from it and put in the cache.

There's a few potential ways to do this:

• Do the basic block decoding inside the perf-dlfilter stub and then pass out the decoded info.
• Use an instruction decoding library from OCaml.
• Use an offline disassembler tool like objdump on the entire binary and fetch basic blocks from that and parse them from text. This is probably the simplest to do from OCaml but would run slowest.