Hello,
if people follow the git repo of ulp they might have noticed that the
consistency model of checking library entries and exits was disabled.
This is to describe the background for that (if for nothing else than us
not forgetting the reasons :-) ).
So, the consistency model basically answers the question "is this
livepatch currently safe to activate for that thread?". There are many
methods that can be designed for this (and it includes the trivial "yes!"
model), but fairly from the beginning we settled on a model that would
track library entries and exits, and only consider live-patch application
safe if the library affected by the live patch was not currently active
(on the call stack).
There are certain conditions and fact that need to be taken into account
for this to work:
* for library entry tracking you need exit tracking
* you need to do the tracking right from process start, even without any
live patches loaded or active; otherwise you might see unbalancend
entry/exits and the library might be considered non-entered even when
it's entered at patch application tyime
* we want the tracking to be per-thread, otherwise one thread might block
application of the live patch for the whole process indefinitely
We do the entry tracking by function entry point redirection (for exported
functions). That's easy. Exit tracking is harder. Without much
toolchain help we can't redirect the return path, so we resorted to frame
stealing:
* in the entry tracker we modify the original return address (on stack
or return register) to point to the exit tracker, and store the original
return address at $place; the exit tracker then restores things and
ultimately returns to the original return address
Now, there are further considerations for $place:
* we can't modify the upward stack: they might hold function arguments
* we can't use downward stack: it will be considered changable by the
called functions
* we can't allocate our own space on stack: function arguments to the
original callee are relative to the stack pointer (or frame pointer
derived from it), so changing the stack pointer would invalidate that.
Even if we consider addresses of stack slots for argument passing to be
undefined we still would need to copy the incoming arguments to a new
place, with the problem that the entry tracker doesn't know _how much_
to copy (in the general case, e.g. variadic functions, only the caller
knows how much space the arguments needed on the stack).
So, we can't have $place be on the stack. But it needs to be per-thread.
So, there's only one possibility: thread-local storage in one or another
way. That's indeed the solution taken by libpulp, it stores the original
return address into some TLS space and all was good.
Well, except that backtraces don't work then. libpulp took precaution to
mark the stolen frames as not backtracable (i.e. as stop points for back
traces), and at least you would see in gdb that the exit tracker was the
top-most frame and be reminded that something special was going on.
But backtraces aren't only a pure debug facility. They are used for
frame unwinding while throwing exceptions and for pthread cancellation, so
backtraces not working over shared library borders also mean exceptions
and thread cancellation not working over shared library borders. That's
of course not acceptable.
So, ideally we would fix backtraces to do work with the entry/exit
tracking. Turns out that this isn't entirely trivial. During the process
of unwinding there is one thing that needs to be done for each unwound
frame, amongst other things: given the current frame and register state,
get the return address of the calling frame. This requires certain pieces
of information about the frames, for which there are multiple
possiblities: windows uses standardized code layout plus custom info
pieces per function, linux most often uses DWARF unwind information to
describe frame layout and return address (arm and aarch64 uses some more
compact form).
The important thing to know is that the unwinder has some internal state
of the current frame and when trying to get the return address for it, it
essentially is given a symbolic expression (on linux a DWARF expression)
that the unwinder interprets to calculate the value. (One example of such
symbolic expression would be: "add 8 to the frame pointer; that's the
place containing the current return address"). This expression comes from
the program itself (e.g. in the .eh_frame section), so the program itself
can specify how the unwinder needs to calculate things according to how
the program was produced.
So, for the stolen frame the return address is stored at some TLS place,
so that's what the DWARF expression for the return address needs to say.
Luckily there are DWARF operations that specify TLS addresses:
DW_OP_GNU_push_tls_address and DW_OP_form_tls_address, i.e. a GNU
extension or DWARF v3, so we can describe the situation we have in the
unwind information.
Very and extremely unfortunately our normal unwinder (in libgcc_s) doesn't
support this operation in its expression interpreter :-/ What's worse,
its structure is basically this:
switch (dwarf->opcode) {
... nothing with tls ...
default: abort();
}
i.e. whenever the current unwinder would hit onto a frame that used the
DWARF TLS opcodes it would abort the program. That's even worse than
unwinding not working for one thread.
Now, we could fix that unwinder (which we'll do). That leaves an unknown
number of other unwinders in the world: they might be statically linked
variants of the libgcc_s unwinder, or they might be completely different
unwinders from unknown sources; but it's reasonable to assume that they
don't implement the TLS opcodes either (no matter if they abort on unknown
opcodes or not). Either way, simply using the dwarf TLS opcodes in our
libraries exposes us to potential and unknown instability due to
non-support in the unwinders, something that live patching is supposed to
protect against ;-)
We have no real solution to the above; a few can be imagined though:
* fix all unwinders to accept TLS opcodes
* make $place not be TLS (seems impossible, but who knows?)
* don't steal frames: exit tracking then needs to be done different, e.g.
by toolchain improvements to also be able to redirect the return paths
of functions
* don't do entry/exit tracking at all
For now we resort to the last of these, ditch entry/exit tracking. This
gives us a much weaker consistency model, and hence more care needs to be
applied when constructing a live patch. We deem that to be okay for now.
We want to make it so that a live patch can request a certain consistency
model, so that in the future a entry/exit tracker, or something completely
different, can be (re)implemented and enlarge the set of possible live
patches.
(One obvious alternative consistency checker would be one looking at
backtraces itself to see if problematic functions are currently active).
There's an advantage to not doing library entry/exit tracking:
performance. The indirection through the tracking code, in particular the
need to use TLS for the original return address and the
"we're-in-that-lib" flag, are not cheap. Together with the fact that lib
entry/exit has to be tracked right from process start makes the
performance impact for userspace live patching noticable (not terribly
so, but measurable). Obviously without such tracking we don't pay that
cost at all. At least something ;-)
Ciao,
Michael.
