Hi all,
I would like to introduce to you an alternative for update-alternatives called libalternatives [1]. The main motivation for myself to work on this is that update-alternatives is both too rigid and too simple to actually get the job done.
To better understand what I'm talking about, first we have to actually understand what update-alternatives is and how it functions. Some of you may have heard that there is more than one console text editor. Some would have vim installed, and some would have emacs installed and some others may even like to work in nano or ed. The problem arose when some 3rd party program (eg. mutt) needed to edit some file (eg. mail) and then there was no simple way of actually starting this editor. Here is where update-alternatives comes in - it allows each of these program to register as an editor and then update-alternatives would install a symlink to the highest priority editor under /usr/bin/editor. The world was simple.
Fast forward a decade or two and today we have various usages of this symlink management system into areas that were never even imagined when it was created. We have symlinks to whole directories. We have multiple versions of python that can be co-installed. And here we come across the first major obstacle and limitation to simply using symlinks -- they only really allow a SINGLE choice to be preferred. Take for example nodejs. You can have nodejs14 and nodejs16 co-installed and running and you could have node pointing to the latter (actually, this is not what happens because precisely of this limitation :-). But, what happens when you run `node14` and then that program re-execs #!/usr/bin/node in a subprocess? The subprocess runs node16 and you are out of luck. You couldn't even compile a binary node module with the older version! What actually happens today is that node stores the major version it executes under in the environment and when /usr/bin/node is executed, it will re-run that version and not the default version -- today update-alternatives still manages node-default symlink.
I expect the same issue exists with ruby and python3 and other interpreters but it's not viewed as a deal breaker but an ugly nuisance. Maybe now this can be improved?
The second issue that comes up is inability to actually manage preferences on a user level. Imagine you have all editors installed on a multi-user system. The user cannot specify their preferences and must bypass update-alternatives and resort to manual symlink hackery in ~/.bin or similar.
The third issue is packaging this mess. How many times have any of you made an error with update-alternatives that then broke an update and became almost impossible to fix without further scriptlet hacks? It's fragile, to say the least. The system is also changed during installation and not at runtime -- this makes it potentially inconsistent between snapshots.
Having said all this, my main motivation for developing an alternative is removing the node-default symlink and moving the logic into a library. The /usr/bin/node would link to it and exec the preferred node version based on user preferences, admin preferences and packager preference, in that order.
libalternatives provides a single executable - /usr/bin/alts. If you run it, you can use it to change the default preferences. If you executable is simple and does not require additional logic, you can just do a symlink,
/usr/bin/editor -> /usr/bin/alts
and preferred editor will be run. If you have additional logic, like /usr/bin/node does, just call execDefault(argc) if and when you want preferred alternative executable exec().
At the moment, I've adjusted the packages of various node versions in my home project - home:adamm:alternatives. There is also an example package there [3] that executes either true or false - I hope the file layout is easy to understand.
I think I could continue with details, but maybe I'll wait for feedback if anyone finds this actually interesting and worthwhile or maybe never even given it thought until now :-)
Cheers, Adam
PS. libalternatives is not yet in Factory but should be submitted here in a few days.
[1] - https://github.com/openSUSE/libalternatives [2] - https://build.opensuse.org/package/show/home:adamm:alternatives/libalternati... [3] - home:adamm:alternatives/true_or_false
On Wed, Jun 2 2021 at 09:40:40 +0200, Adam Majer amajer@suse.de wrote:
The third issue is packaging this mess. How many times have any of you made an error with update-alternatives that then broke an update and became almost impossible to fix without further scriptlet hacks? It's fragile, to say the least. The system is also changed during installation and not at runtime -- this makes it potentially inconsistent between snapshots.
It was suggested to handle this as an abstraction in rpm specs, with update-alternatives being handled with rpmdb, which is probably the method I would like the most https://github.com/rpm-software-management/rpm/issues/993
Scriplets are an awful way to deal with this, I agree, but I think managing this through rpm is a better way to go
LCP [Sasi] https://lcp.world
On Wed, Jun 02, 2021 at 10:21:43AM +0200, Sasi Olin wrote:
On Wed, Jun 2 2021 at 09:40:40 +0200, Adam Majer amajer@suse.de wrote:
The third issue is packaging this mess. How many times have any of you made an error with update-alternatives that then broke an update and became almost impossible to fix without further scriptlet hacks? It's fragile, to say the least. The system is also changed during installation and not at runtime -- this makes it potentially inconsistent between snapshots.
It was suggested to handle this as an abstraction in rpm specs, with update-alternatives being handled with rpmdb, which is probably the method I would like the most https://github.com/rpm-software-management/rpm/issues/993
Scriplets are an awful way to deal with this, I agree, but I think managing this through rpm is a better way to go
Yes, this is a way of avoiding the scriptlet issue but it doesn't address other issues. The issue with inconsistent snapshots remains especially for transactional updates.
It's a little funny how update-alternatives became entrenched and how many packages are using it now.
- Adam
On Wednesday 2021-06-02 09:40, Adam Majer wrote:
symlinks -- they only really allow a SINGLE choice to be preferred.
It so happens that any ordering of elements in a set implies that one is the first.
But, what happens when you run `node14` and then that program re-execs #!/usr/bin/node in a subprocess? The subprocess runs node16 and you are out of luck.
exec is a willfull boundary. (Just like /usr/libexec and /usr/bin.) Programs using these mechanisms _must not_ care about changes in architecture, language, memory layout, etc. nodejs's behavior sounds incredibly stupid.
I expect the same issue exists with ruby and python3 and other interpreters
python3 would hardly exec /usr/bin/python (v2). They're still sane.
The second issue that comes up is inability to actually manage preferences on a user level. Imagine you have all editors installed on a multi-user system. The user cannot specify their preferences bypass update-alternatives and resort to manual symlink hackery in ~/.bin or similar. [The plan is] moving the logic into a library. The /usr/bin/node would link to it and exec the preferred node version based on user preferences
Making /usr/bin/editor a program that painstakingly loads a ton of ELF libraries, parses config files is such a bad idea. Both
export EDITOR=/usr/bin/joe alias editor='$EDITOR' export PAGER=/usr/bin/less
or
PATH="$HOME/bin:$PATH" ln -s /usr/bin/joe ~/bin/editor
exec the "right" program, "right away" (after perhaps a search through PATH - it's still a _lot_ less syscalls).
Am 02.06.21 um 10:26 schrieb Jan Engelhardt:
python3 would hardly exec /usr/bin/python (v2). They're still sane.
Many ignorant python scripts not strictly adhering to the PEP 394 recommendations [1] unfortunately do. /usr/bin/python pointing to python v2 instead of python3 is a distro choice, not a Python convention. There is a reason why we have thousands of "replace the script interpreter" sed calls in our python package specfiles.
[1] https://www.python.org/dev/peps/pep-0394/#for-python-script-publishers
On Wed, Jun 02, 2021 at 10:50:11AM +0200, Jan Engelhardt wrote:
On Wednesday 2021-06-02 10:45, Ben Greiner wrote:
Am 02.06.21 um 10:26 schrieb Jan Engelhardt:
python3 would hardly exec /usr/bin/python (v2). They're still sane.
Many ignorant python scripts
Ignorant scripts are one thing. Ignorant node runtimes is another level.
Well, I kind of detect a bias? Here's a naive python example,
f133:~ # cat worker.py #!/usr/bin/python3
import platform print(platform.python_version())
f133:~ # cat daemon.py #!/usr/bin/python3
import subprocess
subprocess.Popen(['./worker.py']).wait()
f133:~ # python3. python3.6 python3.6m python3.8 python3.9 f133:~ # python3.6 ./daemon.py 3.8.10 f133:~ # python3.6 ./worker.py 3.6.13
So, with node, if you did this experiment, and you started it with node10, you would get node10 in the worker irrespective what /usr/bin/node points to.
Basically, once we allow multiple interpreters to be co-installed, they actually should run as expected. No? Saying the problem is "out of scope" tells me that something is abused for purposes that were not intended (looking at update-alternatives here).
Anyway, this is an example of why pure symlinks are bad and actually result in confusion at runtime.
- Adam
Am 02.06.21 um 14:52 schrieb Adam Majer:
On Wed, Jun 02, 2021 at 10:50:11AM +0200, Jan Engelhardt wrote:
On Wednesday 2021-06-02 10:45, Ben Greiner wrote:
Am 02.06.21 um 10:26 schrieb Jan Engelhardt:
python3 would hardly exec /usr/bin/python (v2). They're still sane.
Many ignorant python scripts
Ignorant scripts are one thing. Ignorant node runtimes is another level.
Well, I kind of detect a bias? Here's a naive python example,
f133:~ # cat worker.py #!/usr/bin/python3
import platform print(platform.python_version())
f133:~ # cat daemon.py #!/usr/bin/python3
import subprocess
subprocess.Popen(['./worker.py']).wait()
f133:~ # python3. python3.6 python3.6m python3.8 python3.9 f133:~ # python3.6 ./daemon.py 3.8.10 f133:~ # python3.6 ./worker.py 3.6.13
So, with node, if you did this experiment, and you started it with node10, you would get node10 in the worker irrespective what /usr/bin/node points to.
Basically, once we allow multiple interpreters to be co-installed, they actually should run as expected.
But how do you know what "is expected"? In the Python case we can only assume that PEP 394 is the de-facto standard:
---PEP 394--- - When reinvoking the interpreter from a Python script, querying sys.executable to avoid hardcoded assumptions regarding the interpreter location remains the preferred approach. ---PEP 394---
No? Saying the problem is "out of scope" tells me that something is abused for purposes that were not intended (looking at update-alternatives here).> Anyway, this is an example of why pure symlinks are bad and actually result in confusion at runtime.
Your example does not involve u-a at all. We don't put the python3 command under u-a control. We even go out of our way and fix the "ignorant scripts", who install wrong script interpreter lines and do something to the extent of replacing `subprocess.call("./worker.py")` or `subprocess.call("python", "./worker.py")` with `subprocess.call(sys.executable, "./worker.py")`.
Am 02.06.21 um 10:46 schrieb Adam Majer:
python3 exec python3, of course. But what happens if you have python3.8 and python3.10 installed and want your app to use python3.8? That app then starts a child process and you are in python3.10 then?
If you know you want python3.8, then call python3.8. If the app depends on the same python version as it is called on the toplevel call, it MUST use `sys.executable`. `"/usr/bin/python3"` is always the current TW primary python3 flavor and all dependencies for the script calling it must be provided in that flavor.
When Python setuptools installs "entrypoints" it takes care of the correct script interpreter automatically. In the case of multiple flavors, this is overwritten by `%python_clone -a %{buildroot}%{_bindir}/myentrypoint` creating the u-a symlinks and adjusting the script interpreter lines.
Any python package, which installs commands into %{_bindir} must either be for the primary python3 flavor only or the handling of different flavors must be taken care of by u-a and adjusted script interpreter lines as described above.
Apparently Node.js is more lenient on different flavors. But as you describe, this calls for problems with u-a.
- Ben
On Wed, Jun 2, 2021 at 2:22 PM Ben Greiner code@bnavigator.de wrote:
Am 02.06.21 um 14:52 schrieb Adam Majer:
On Wed, Jun 02, 2021 at 10:50:11AM +0200, Jan Engelhardt wrote:
On Wednesday 2021-06-02 10:45, Ben Greiner wrote:
Am 02.06.21 um 10:26 schrieb Jan Engelhardt:
python3 would hardly exec /usr/bin/python (v2). They're still sane.
Many ignorant python scripts
Ignorant scripts are one thing. Ignorant node runtimes is another level.
Well, I kind of detect a bias? Here's a naive python example,
f133:~ # cat worker.py #!/usr/bin/python3
import platform print(platform.python_version())
f133:~ # cat daemon.py #!/usr/bin/python3
import subprocess
subprocess.Popen(['./worker.py']).wait()
f133:~ # python3. python3.6 python3.6m python3.8 python3.9 f133:~ # python3.6 ./daemon.py 3.8.10 f133:~ # python3.6 ./worker.py 3.6.13
So, with node, if you did this experiment, and you started it with node10, you would get node10 in the worker irrespective what /usr/bin/node points to.
Basically, once we allow multiple interpreters to be co-installed, they actually should run as expected.
But how do you know what "is expected"? In the Python case we can only assume that PEP 394 is the de-facto standard:
---PEP 394---
- When reinvoking the interpreter from a Python script, querying
sys.executable to avoid hardcoded assumptions regarding the interpreter location remains the preferred approach. ---PEP 394---
No? Saying the problem is "out of scope" tells me that something is abused for purposes that were not intended (looking at update-alternatives here).> Anyway, this is an example of why pure symlinks are bad and actually result in confusion at runtime.
Your example does not involve u-a at all. We don't put the python3 command under u-a control. We even go out of our way and fix the "ignorant scripts", who install wrong script interpreter lines and do something to the extent of replacing `subprocess.call("./worker.py")` or `subprocess.call("python", "./worker.py")` with `subprocess.call(sys.executable, "./worker.py")`.
Am 02.06.21 um 10:46 schrieb Adam Majer:
python3 exec python3, of course. But what happens if you have python3.8 and python3.10 installed and want your app to use python3.8? That app then starts a child process and you are in python3.10 then?
If you know you want python3.8, then call python3.8. If the app depends on the same python version as it is called on the toplevel call, it MUST use `sys.executable`. `"/usr/bin/python3"` is always the current TW primary python3 flavor and all dependencies for the script calling it must be provided in that flavor.
When Python setuptools installs "entrypoints" it takes care of the correct script interpreter automatically. In the case of multiple flavors, this is overwritten by `%python_clone -a %{buildroot}%{_bindir}/myentrypoint` creating the u-a symlinks and adjusting the script interpreter lines.
Any python package, which installs commands into %{_bindir} must either be for the primary python3 flavor only or the handling of different flavors must be taken care of by u-a and adjusted script interpreter lines as described above.
Apparently Node.js is more lenient on different flavors. But as you describe, this calls for problems with u-a.
Why are we using alternatives with Python anyway? Why not just use the swappable package strategy for `/usr/bin/python` and `/usr/bin/python3`?
On Wed, Jun 02, 2021 at 10:26:24AM +0200, Jan Engelhardt wrote:
On Wednesday 2021-06-02 09:40, Adam Majer wrote:
symlinks -- they only really allow a SINGLE choice to be preferred.
It so happens that any ordering of elements in a set implies that one is the first.
But, what happens when you run `node14` and then that program re-execs #!/usr/bin/node in a subprocess? The subprocess runs node16 and you are out of luck.
exec is a willfull boundary. (Just like /usr/libexec and /usr/bin.) Programs using these mechanisms _must not_ care about changes in architecture, language, memory layout, etc. nodejs's behavior sounds incredibly stupid.
Well, it needs to exec g++ and python3 - that's the GYP, build system. This in turns needs to find parameters, which is from node process and you end up with wrong one.
I expect the same issue exists with ruby and python3 and other interpreters
python3 would hardly exec /usr/bin/python (v2). They're still sane.
python3 exec python3, of course. But what happens if you have python3.8 and python3.10 installed and want your app to use python3.8? That app then starts a child process and you are in python3.10 then?
The second issue that comes up is inability to actually manage preferences on a user level. Imagine you have all editors installed on a multi-user system. The user cannot specify their preferences bypass update-alternatives and resort to manual symlink hackery in ~/.bin or similar. [The plan is] moving the logic into a library. The /usr/bin/node would link to it and exec the preferred node version based on user preferences
Making /usr/bin/editor a program that painstakingly loads a ton of ELF libraries, parses config files is such a bad idea. Both
export EDITOR=/usr/bin/joe alias editor='$EDITOR' export PAGER=/usr/bin/less
or
PATH="$HOME/bin:$PATH" ln -s /usr/bin/joe ~/bin/editor
exec the "right" program, "right away" (after perhaps a search through PATH - it's still a _lot_ less syscalls).
Not sure editor cares about syscall overhead. But yes, there is about 3 or 4 syscalls + the exec(). Starting actual interpreters like bash is a lot more overhead.
- Adam
Adam Majer wrote:
On Wed, Jun 02, 2021 at 10:26:24AM +0200, Jan Engelhardt wrote:
[...] Making /usr/bin/editor a program that painstakingly loads a ton of ELF libraries, parses config files is such a bad idea. Both
export EDITOR=/usr/bin/joe alias editor='$EDITOR' export PAGER=/usr/bin/less
or
PATH="$HOME/bin:$PATH" ln -s /usr/bin/joe ~/bin/editor
exec the "right" program, "right away" (after perhaps a search through PATH - it's still a _lot_ less syscalls).
Not sure editor cares about syscall overhead. But yes, there is about 3 or 4 syscalls + the exec(). Starting actual interpreters like bash is a lot more overhead.
So what's the verdict here? Is libalternative suitable to replace update-alternatives universally also for stuff like /bin/sh?
cu Ludwig
Hi ,
I noticed quite a bunch of submit requests to python packages such as https://build.opensuse.org/request/show/919833 (pip) and https://build.opensuse.org/request/show/919826 (ipython)
For the sake of sanity. Please don't add Yet Another Way of Flavor Switching to the Python world on openSUSE distros, especially not in a way that makes the specfiles even more complicated than they already are. The python-rpm-macros support for update-alternatives works. It is not great, but it is the best thing we have so far.
Am 02.06.21 um 09:40 schrieb Adam Majer:
Hi all,
[...]
The second issue that comes up is inability to actually manage preferences on a user level. Imagine you have all editors installed on a multi-user system. The user cannot specify their preferences and must bypass update-alternatives and resort to manual symlink hackery in ~/.bin or similar.
Python users are recommended to use virtual environments for such preferences on the user level. Adding yet another distribution specific way does not help to clear up the jungle.
- Ben
Dne 19. 09. 21 v 14:19 Ben Greiner napsal(a):
I noticed quite a bunch of submit requests to python packages such as https://build.opensuse.org/request/show/919833 (pip) and https://build.opensuse.org/request/show/919826 (ipython)
For the sake of sanity. Please don't add Yet Another Way of Flavor Switching to the Python world on openSUSE distros, especially not in a way that makes the specfiles even more complicated than they already are. The python-rpm-macros support for update-alternatives works. It is not great, but it is the best thing we have so far.
I have previously accepted number of these request and I am sorry, it was a mistake. All I was able to find, at least in d:l:p were reverted and their counterparts to openSUSE:Factory revoked.
Stefan, please, suggest an improvement in https://github.com/openSUSE/python-rpm-macros/issues which would cleanly hide whole libalternative thing behind our current macros (if possible).
It is not the reason, but all those requests also failed with errors like https://da.gd/hFPa0.
Best,
Matěj
-
Adam Majer -
Ben Greiner -
Jan Engelhardt -
Ludwig Nussel -
Matěj Cepl -
Neal Gompa -
Sasi Olin