Attempt to build a "hello world" live patch

Hello, I've tried to build a "hello world" demo live patch for a simple application. I focused on minimal code and applied a "cargo-cult" approach. First, I built an applicaiton with a shared library. ------------8<------------ main.c: #include <unistd.h> extern void workload(); int main(void) { int counter; for(counter = 0; counter < 30; counter++) { workload(); sleep(2); } } workload.c: #include <stdio.h> void workload(void) { printf("Hello World!\n"); } Compiled as: $ gcc -shared -fPIC -fpatchable-function-entry=40,38 -I/data/src/libpulp/include -o libworkload.so /data/src/libpulp/lib/trm.S workload.c $ /data/src/libpulp/tools/ulp_post libworkload.so $ gcc -Wl,-rpath,. -L. -o main main.c -lworkload And it works: $ ./main Hello World! Hello World! ^C ------------>8------------ Then I prepared a patch for workload(). ------------8<------------ workload_patch.c: #include <stdio.h> void workload_modernized(void) { printf("hello, world\n"); } libworkload_patch.dsc: /data/src/libpulp/demo/libworkload_patch.so @/data/src/libpulp/demo/libworkload.so workload:workload_modernized Compiled as: $ gcc -shared -fPIC -o libworkload_patch.so workload_patch.c $ /data/src/libpulp/tools/ulp_packer libworkload_patch.dsc libworkload_patch.ulp ------------>8------------ At this point, I believe I have all the bits in place and I want to try live patching of 'main'. ------------8<------------ $ LD_PRELOAD=/data/src/libpulp/lib/.libs/libpulp.so ./main & [1] 27910 libpulp loaded... Hello World! $ /data/src/libpulp/tools/ulp_trigger "$(pidof main)" /data/src/libpulp/demo/libworkload_patch.ulp ulp: to be patched object (/data/src/libpulp/demo/libworkload.so) not loaded. ------------>8------------ What am I doing wrong, apart from the uneducated approach, that ulp_trigger complaints about the missing libworkload.so? Libor Side notes: - It is unclear how to build live patches. README.md contains the high-level overview but not concrete steps or a pointer to a "how to". - ulp_packer help is wrong. It says "packer <descr> <.so> [.ulp]" while it's now "packer <descr> <.ulp>" - The role of ulp_post is unclear in the process. There is a clue in the commit log that introduces it but it was beyong my current knowledge level. - I've inferred compiler parameters and command usage from what I saw in "make check" output. - ulp_trigger says nothing in case libpulp.so is not preloaded. I suggest that it prints some diagnostic message. -- Libor Pechacek SUSE Labs Remember to have fun...

Hi, Libor... You did everything right... On Wed, 10 Mar 2021, Libor Pechacek wrote:
$ gcc -Wl,-rpath,. -L. -o main main.c -lworkload
^ ^ ... it is very unfortunate that you hit a usability bug. It would have worked if you had used $PWD instead of '.', such as: $ gcc -Wl,-rpath,$PWD -L$PWD -o main main.c -lworkload Libpulp, more specifically ulp_trigger, currently ignores things that don't start with a foward slash. This is where it happens: https://github.com/SUSE/libpulp/blob/master/tools/introspection.c#L384-L385 These lines should probably just go away (probably no side-effects).
What am I doing wrong, apart from the uneducated approach, that ulp_trigger complaints about the missing libworkload.so?
Nothing wrong. I'm sorry you hit this bug.
Side notes: - It is unclear how to build live patches. README.md contains the high-level overview but not concrete steps or a pointer to a "how to". - ulp_packer help is wrong. It says "packer <descr> <.so> [.ulp]" while it's now "packer <descr> <.ulp>" - The role of ulp_post is unclear in the process. There is a clue in the commit log that introduces it but it was beyong my current knowledge level. - I've inferred compiler parameters and command usage from what I saw in "make check" output. - ulp_trigger says nothing in case libpulp.so is not preloaded. I suggest that it prints some diagnostic message.
Thank you so much for this feedback! Cheers, Gabrie

On St 10-03-21 14:29:00, Gabriel F. T. Gomes wrote:
Hi, Libor...
You did everything right...
On Wed, 10 Mar 2021, Libor Pechacek wrote:
$ gcc -Wl,-rpath,. -L. -o main main.c -lworkload
^ ^ ... it is very unfortunate that you hit a usability bug. It would have worked if you had used $PWD instead of '.', such as:
$ gcc -Wl,-rpath,$PWD -L$PWD -o main main.c -lworkload
Indeed! Now it works! :)
Libpulp, more specifically ulp_trigger, currently ignores things that don't start with a foward slash. This is where it happens:
https://github.com/SUSE/libpulp/blob/master/tools/introspection.c#L384-L385
These lines should probably just go away (probably no side-effects).
I'm not sure. There is a good reason why "." should not be in $PATH. IMHO the same reasoning applies to -rpath as well. Anyway, trying to invoke 'main' from another directory fails, so I'd consider my -rpath setting flawed. I agree, however, that a warning from ulp_trigger would be useful when it encounters a library loaded from relative path. Alternativelu, it may have an option to enable patching libraries with relative paths, should anyone depend on such behavior. Thanks! Libor -- Libor Pechacek SUSE Labs Remember to have fun...

On Čt 11-03-21 11:21:40, Libor Pechacek wrote:
On St 10-03-21 14:29:00, Gabriel F. T. Gomes wrote:
Hi, Libor...
You did everything right...
On Wed, 10 Mar 2021, Libor Pechacek wrote:
$ gcc -Wl,-rpath,. -L. -o main main.c -lworkload
^ ^ ... it is very unfortunate that you hit a usability bug. It would have worked if you had used $PWD instead of '.', such as:
$ gcc -Wl,-rpath,$PWD -L$PWD -o main main.c -lworkload
Indeed! Now it works! :)
Nicolai reminded me about the magic $ORIGIN token. The demo works even better with that one. Encouraged by the success, I've entertained the idea of getting rid of the LD_PRELOAD. So, I tried: $ gcc -Wl,-rpath,\$ORIGIN -Wl,-rpath,/data/src/libpulp/lib/.libs -L. \ -L ../lib/.libs/ -o main main.c -lworkload -lpulp And, voilà! I can just run "main" and live patch it without any extra hassle. Before I start exploring this possibility further, do you think that it's OK to leverage the dynamic linker for preloading libpulp? I can imagine users may want to build "live patchable" applications, and perhaps also patchable shared libraries. In that can thay may want reference libpulp from the shared library. Thoughts? Libor -- Libor Pechacek SUSE Labs Remember to have fun...

Hi, Libor, On Fri, 12 Mar 2021, Libor Pechacek wrote:
Nicolai reminded me about the magic $ORIGIN token. The demo works even better with that one.
TIL $ORIGIN... Thanks! :)
Encouraged by the success, I've entertained the idea of getting rid of the LD_PRELOAD. So, I tried:
$ gcc -Wl,-rpath,\$ORIGIN -Wl,-rpath,/data/src/libpulp/lib/.libs -L. \ -L ../lib/.libs/ -o main main.c -lworkload -lpulp
And, voilà! I can just run "main" and live patch it without any extra hassle.
Before I start exploring this possibility further, do you think that it's OK to leverage the dynamic linker for preloading libpulp?
Do you mean the use of "-lpulp", so you get rid of LD_PRELOAD? I don't see any problems with it, though I should add that ld's -as-needed flag (which is used by default in OBS builds) would break that, because your library (or any to-be-made-livepatchable library, for that matter) doesn't actually depend on libpulp.so; thus, you might need to use: -Wl,-no-as-needed -lpulp. I'm not sure what's the main purpose of using LD_PRELOAD. The only advantages (as compared to hard-coding libpulp.so.N into NEEDED entries in the Elf file) I can think of are: 1. one less package on your system if you don't want live-patching; 2. Being able to selectively enable live-patches on a process-by-process basis.
I can imagine users may want to build "live patchable" applications, and perhaps also patchable shared libraries. In that can thay may want reference libpulp from the shared library.
I think I see what you mean... You would like to be able to create a package with a library that is live-patchable out-of-the-box, right? For instance, if someone installs your package for libworkload, then all projects linked against it will produce live-patchable applications which, when executed, start live-patchable processes (even if LD_PRELOAD is not used). Is that what you mean? If so, I don't know how end-users want libpulp to behave... Not yet. Your use case is a good hint for me, though... Initially, I thought everybody would prefer the LD_PRELOAD approach, so the package at OBS (home:ULP) doesn't even distributes libpulp.so in the development package (only libpulp.so.0 on the library package), as explained in the spec file: %install (...) # Remove .la and .so files. libpulp.so is not supposed to be linked # against any programs or libraries, but LD_PRELOAD'ed, so do not # distribute it, not even in the devel package. find %{buildroot}/%{_prefix} -name libpulp.la -delete find %{buildroot}/%{_prefix} -name libpulp.so -delete (lines 92-96 in https://build.opensuse.org/package/view_file/home:ULP/libpulp/libpulp.spec?e...) Perhaps that should go away, so that users can do like you and use -lpulp. Cheers, Gabriel

On Pá 12-03-21 14:28:50, Gabriel F. T. Gomes wrote: [...]
Before I start exploring this possibility further, do you think that it's OK to leverage the dynamic linker for preloading libpulp?
Do you mean the use of "-lpulp", so you get rid of LD_PRELOAD?
Yes. Sorry for leaving out the details.
I don't see any problems with it, though I should add that ld's -as-needed flag (which is used by default in OBS builds) would break that, because your library (or any to-be-made-livepatchable library, for that matter) doesn't actually depend on libpulp.so; thus, you might need to use: -Wl,-no-as-needed -lpulp.
Understood. Thanks for the hint!
I'm not sure what's the main purpose of using LD_PRELOAD. The only advantages (as compared to hard-coding libpulp.so.N into NEEDED entries in the Elf file) I can think of are: 1. one less package on your system if you don't want live-patching; 2. Being able to selectively enable live-patches on a process-by-process basis.
I don't know the history of the LD_PRELOAD thing either. I suspect, however, that no one evaluated the options so far. :)
I can imagine users may want to build "live patchable" applications, and perhaps also patchable shared libraries. In that can thay may want reference libpulp from the shared library.
I think I see what you mean... You would like to be able to create a package with a library that is live-patchable out-of-the-box, right? For instance, if someone installs your package for libworkload, then all projects linked against it will produce live-patchable applications which, when executed, start live-patchable processes (even if LD_PRELOAD is not used). Is that what you mean?
Exactly. Complicated start scripts of some applications (e.g., Firefox has a shell script to start the main process, Android Studio has similar shell magic) brought me to the idea. It might be complicated to stick an LD_PRELOAD into the start-up mechanism, or it might be outright impossible to do so when the native binary is called from, e.g., a proprietary Java code. At that point, I imagined a user who would like to live-patch a certain shared library, which is in use by a complicated application, without changing the application itself. Not that someone has asked. I'm merely presuming the demand.
If so, I don't know how end-users want libpulp to behave... Not yet.
Your use case is a good hint for me, though... Initially, I thought everybody would prefer the LD_PRELOAD approach, so the package at OBS (home:ULP) doesn't even distributes libpulp.so in the development package (only libpulp.so.0 on the library package), as explained in the spec file:
%install (...) # Remove .la and .so files. libpulp.so is not supposed to be linked # against any programs or libraries, but LD_PRELOAD'ed, so do not # distribute it, not even in the devel package. find %{buildroot}/%{_prefix} -name libpulp.la -delete find %{buildroot}/%{_prefix} -name libpulp.so -delete
(lines 92-96 in https://build.opensuse.org/package/view_file/home:ULP/libpulp/libpulp.spec?e...)
Perhaps that should go away, so that users can do like you and use -lpulp.
I can imagine that someone would even want to statically link libpulp into their app. But we are at the beginning and I'm merely exploring the possibilities. Thanks for your insights, Gabriel! Libor -- Libor Pechacek SUSE Labs Remember to have fun...

Hello, On Fri, 12 Mar 2021, Libor Pechacek wrote:
Indeed! Now it works! :)
Nicolai reminded me about the magic $ORIGIN token. The demo works even better with that one.
Encouraged by the success, I've entertained the idea of getting rid of the LD_PRELOAD. So, I tried:
$ gcc -Wl,-rpath,\$ORIGIN -Wl,-rpath,/data/src/libpulp/lib/.libs -L. \ -L ../lib/.libs/ -o main main.c -lworkload -lpulp
And, voilà! I can just run "main" and live patch it without any extra hassle.
Of course, that's nothing to do with $ORIGIN, though. You linked against libpulp, hence it's loaded, hence it can be used by the ulp tool to live patch the process. The problem is the following: imagine you have a live patchable libfoo, and a random collection of binaries linking against libfoo. (To see the size of that random set, imagine libfoo being libc.so.6). Do you want to make all these binaries live patchable (or to be precise: processes resulting from execve(2) of these binaries) simply by installing the live patchable libfoo? Even if enabling live patching for a process induces a performance hit? Even if you decide to say "yes" to this question: do you want to have to link all these binaries against libpulp (which comes with the problem that suddenly those binaries only work when libpulp is installed, even though they themself don't make use of any live-patching facilities). You could also link a live patchable library against libpulp (and so make any process using that library also be live patchable). But that has some interesting consequences about entry tracking: it starts globally for a process only when libpulp is loaded, so anything loaded before libpulp is loaded isn't tracked. This might not be a problem, as normally it should be only required to track stuff in the live-patchable libs, but we punted on that. So, for now we decided to let the answer to the question "should a process be live patchable?" be a policy decision that is expressed through environment variables. I don't know of a good solution to some of the above sub questions :-/ (Hint: you can force preloading a library to all processes ever started by using /etc/ld.so.preload) Ciao, Michael.

On Po 15-03-21 15:39:48, Michael Matz wrote:
On Fri, 12 Mar 2021, Libor Pechacek wrote: [..]
Encouraged by the success, I've entertained the idea of getting rid of the LD_PRELOAD. So, I tried:
$ gcc -Wl,-rpath,\$ORIGIN -Wl,-rpath,/data/src/libpulp/lib/.libs -L. \ -L ../lib/.libs/ -o main main.c -lworkload -lpulp
And, voilà! I can just run "main" and live patch it without any extra hassle.
Of course, that's nothing to do with $ORIGIN, though. You linked against libpulp, hence it's loaded, hence it can be used by the ulp tool to live patch the process.
Yes. I realize that my message was too brief and there were too few words between neighboring ideas. :) My intent is exactly what you describe. The only information bit I withdrew is that when I link libworkload.so with libpulp.so (gcc -shared ... workload.c -lpulp), then the 'main' process aborts with weird messages right after start on my system.
The problem is the following: imagine you have a live patchable libfoo, and a random collection of binaries linking against libfoo. (To see the size of that random set, imagine libfoo being libc.so.6). Do you want to make all these binaries live patchable (or to be precise: processes resulting from execve(2) of these binaries) simply by installing the live patchable libfoo? Even if enabling live patching for a process induces a performance hit? Even if you decide to say "yes" to this question: do you want to have to link all these binaries against libpulp (which comes with the problem that suddenly those binaries only work when libpulp is installed, even though they themself don't make use of any live-patching facilities).
Agreed. This is the distro vendor view. I imagined an ISV who want to live patch their own shared library in their own set of applications. For such a user, shotgun approach might be valid.
You could also link a live patchable library against libpulp (and so make any process using that library also be live patchable). But that has some interesting consequences about entry tracking: it starts globally for a process only when libpulp is loaded, so anything loaded before libpulp is loaded isn't tracked. This might not be a problem, as normally it should be only required to track stuff in the live-patchable libs, but we punted on that.
Good point. Thanks for the remark!
So, for now we decided to let the answer to the question "should a process be live patchable?" be a policy decision that is expressed through environment variables.
Makes sense. As I wrote in my other message, I'm exploring the possibilities and evaluating the options. This one was moticated by the hypothetical situation in which LD_PRELOAD cannot be propagated though some proprietary and/or complicated start-up code. Thanks for your insights.
I don't know of a good solution to some of the above sub questions :-/
(Hint: you can force preloading a library to all processes ever started by using /etc/ld.so.preload)
Aieee. ;-) Libor -- Libor Pechacek SUSE Labs Remember to have fun...

On Mon, 15 Mar 2021, Michael Matz wrote:
You could also link a live patchable library against libpulp (and so make any process using that library also be live patchable). But that has some interesting consequences about entry tracking: it starts globally for a process only when libpulp is loaded, so anything loaded before libpulp is loaded isn't tracked.
Hmm, perhaps this changed in the past, but since I joined the project, entrance tracking happens regardless of libpulp.so.0 being loaded or not. Entrance tracking is enabled on any library that has been made live-patchable, because the .dynsym section in them is tweaked (by ulp_post (formerly known as dynsym_gate)) so that the addresses of actual functions are replaced with the address of trampolines (added by ulp_post). Just for the sake of completeness, each function has its own trampoline, which stores the address of the actual target function on the stack, then jumps to the entrance tracking routine. By the way, this is the main source of overhead, so having a live-patchable library installed in default locations (or preferred by ld.so.cache) has an effect on the whole system. Perhaps this needs changing (in .spec files, that is).

Hello, On Tue, 16 Mar 2021, Gabriel F. T. Gomes wrote:
You could also link a live patchable library against libpulp (and so make any process using that library also be live patchable). But that has some interesting consequences about entry tracking: it starts globally for a process only when libpulp is loaded, so anything loaded before libpulp is loaded isn't tracked.
Hmm, perhaps this changed in the past, but since I joined the project, entrance tracking happens regardless of libpulp.so.0 being loaded or not.
Oh right. God, I really need to read the code again :-) The ulp_entry still does check if __ulp_global_universe is there or not (which is what indicates the presence or absence of libpulp), but it always does the tracking unconditionally. Hmm. I'm not sure we want to change this right now, or at all. But initially the idea was to only take the big cost hit for processes where libpulp is loaded. That still would need one memory access per entry point (to check for the weak symbol), but at least not go through the whole TLS-address business and memory writes for the counter updates.
Entrance tracking is enabled on any library that has been made live-patchable, because the .dynsym section in them is tweaked (by ulp_post (formerly known as dynsym_gate)) so that the addresses of actual functions are replaced with the address of trampolines (added by ulp_post). Just for the sake of completeness, each function has its own trampoline, which stores the address of the actual target function on the stack, then jumps to the entrance tracking routine.
By the way, this is the main source of overhead, so having a live-patchable library installed in default locations (or preferred by ld.so.cache) has an effect on the whole system. Perhaps this needs changing (in .spec files, that is).
Ciao, Michael.
participants (3)
-
Gabriel F. T. Gomes
-
Libor Pechacek
-
Michael Matz