[opensuse-factory] Can we assume that /bin/sh is bash?
Hi Jan, all, I'd like to carry our OBS dicussion to a wider audience (https://build.opensuse.org/request/show/672510). The question is whether we can assume that "/bin/sh" links to bash, in particular whether rpm scriptlets without explicit -p interpreter can be assumed to interpreted by bash. I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it is not. Unless it's tampered with, /bin/sh is a symlink to /bin/bash. bash is not started in full POSIX mode if invoked as /bin/sh, and even if it's in POSIX mode, it supports some extensions over the POSIX shell spec (e.g. the [[ ]] construct), which makes it behave differently than another shell not supporting [[ ]] would (*). Problably there are more differences, I can't claim to know them all. Here are some arguments why I think it'd be reasonable to assume that /bin/sh is bash on openSUSE: 1. patterns-base-minimal_base depends on bash, and the /bin/sh symlink is a non-configurable part of the "bash" package. 2. we could handle /bin/sh via /etc/alternatives, but we don't. 3. our Wiki suggests testing failing scriplets using "bash -xv" (https://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets) 4. /bin/sh has pointed to bash for a long time (not sure how long exactly). FTR, Fedora basically guarantees (sh == bash) for the purpose of rpm scriptlets (https://fedoraproject.org/wiki/Packaging:Scriptlets). So Fedora <-> openSUSE portability may also be an issue to consider. If we can't assume that /bin/sh is bash, what else can we assume? I recall from earlier work that writing really 100% compatible shell code for all kinds of environments is really hard. E.g., "[" isn't 100% portable either, even though it's part of the POSIX "test" standard (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html). We should have clear rules which syntax expressions can be used in rpm scriptlets, and which can't. IMO we should define one of the various existing shells as a reference by saying "if it's supported by this shell, it's valid scriptlet code". That'd be easier (especially for testing compliance) than referring to a spec. That reference shell doesn't have to be bash, of course. Thanks Martin (*) For example: # ln -s /usr/bin/echo '/usr/bin/[[' # ls -l /bin/sh lrwxrwxrwx 1 root root 4 Feb 8 10:38 /bin/sh -> bash # /bin/sh --posix -c '[[ 2 = 1 ]] && echo yes' # dash -c '[[ 2 = 1 ]] && echo yes' 2 -eq 1 ]] yes -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Martin Wilck <mwilck@suse.com> wrote:
Hi Jan, all,
I'd like to carry our OBS dicussion to a wider audience (https://build.opensuse.org/request/show/672510).
The question is whether we can assume that "/bin/sh" links to bash, in particular whether rpm scriptlets without explicit -p interpreter can be assumed to interpreted by bash.
I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it
UNIX used to have the Bourne Shell installed as /bin/sh, but I suspect that the Bourne Shell is not available on SuSe - or did you start to support "obosh" binaries recently? POSIX requires a platform to explicitly explain how to start a POSIX compliant environment. If this is not available the following method is described.... This typically happens if you call: PATH=`getconf PATH` sh since it is assumed that there is a special PATH with (by default) 100% POSIX compliant binaries (e.g. /usr/xpg4/bin or today /usr/xpg7/bin) and "getconf PATH" returns something like: /usr/xpg4/bin:/usr/ccs/bin:/usr/bin
is not. Unless it's tampered with, /bin/sh is a symlink to /bin/bash. bash is not started in full POSIX mode if invoked as /bin/sh, and even if it's in POSIX mode, it supports some extensions over the POSIX shell spec (e.g. the [[ ]] construct), which makes it behave differently than another shell not supporting [[ ]] would (*). Problably there are more differences, I can't claim to know them all.
Correct, [[ ... ]] is intentionally not part of the POSIX standard.
If we can't assume that /bin/sh is bash, what else can we assume? I recall from earlier work that writing really 100% compatible shell code for all kinds of environments is really hard. E.g., "[" isn't 100% portable either, even though it's part of the POSIX "test" standard (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html).
This is not correct, "test" is 100% portable if your scripts follow the POSIX rules. Note that you need to use only simple expressions as POSIX defines the behavior for up to 4 argument to "test" only.
We should have clear rules which syntax expressions can be used in rpm scriptlets, and which can't. IMO we should define one of the various existing shells as a reference by saying "if it's supported by this shell, it's valid scriptlet code". That'd be easier (especially for testing compliance) than referring to a spec. That reference shell doesn't have to be bash, of course.
Thanks Martin
(*) For example:
# ln -s /usr/bin/echo '/usr/bin/[[' # ls -l /bin/sh lrwxrwxrwx 1 root root 4 Feb 8 10:38 /bin/sh -> bash # /bin/sh --posix -c '[[ 2 = 1 ]] && echo yes'
This is definitely not portable at all, as --posix is not in the POSIX standard. This is the reason, why it is better to implement 100% POSIX compliant binaries in a special install path, as not even "-o posix" is defined by the POSIX standard.
# dash -c '[[ 2 = 1 ]] && echo yes' 2 -eq 1 ]] yes
dash prints: dash: 1: [[: not found Note that many Linux platforms today come with "dash" installed as /bin/sh but dash is not POSIX compliant as dash does not support multi-byte characters and POSIX only allows to omit the so called "XSI" support when on small embedded platforms. If you like to use something that is POSIX compliant and gives a similar set of features as dash, I recommend to use "pbosh", a minimal POSIX variant of the recent Bourne Shell development version. This shell has been verified to work perfectly on Gentoo Linux when installed as /bin/sh. Using "pbosh" makes sense since it is POSIX, supports multi-byte characters and is even a bit faster than dash. BTW: another problem with POSIX compliance related to bash is that bash by default implements a non POSIX "echo" builtin command. The bash binaries delivered with Apple and Solaris use a compile variant with POSIX compliant "echo". Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 2019-02-08 13:22, Martin Wilck wrote:
I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it is not.
But it could be. In fact, it can be. `zypper dup` yesterday with an update block of 120 packages yielded only 7 scriptlet problems when sh is dash.
If we can't assume that /bin/sh is bash, what else can we assume? I recall from earlier work that writing really 100% compatible shell code for all kinds of environments is really hard.
No one needs to write sh-compatible code. The code just needs to be annotated that it requires a certain interpreter. Since script files already have a #! line most of the time, that's taken care of. That leaves script "snippets" that rely on implicit interpreters, such as in .spec. Since I literally get to see every new line of spec code due to factory-review, many specfiles have already gotten their treatment (if one was even needed). The upside is that specfiles don't need a lot of work because everyone tries to keep them short anyway and reuse macros due to the volume of it all. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 2019-02-08 14:16, Joerg Schilling wrote:
I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it
UNIX used to have the Bourne Shell installed as /bin/sh, but I suspect that the Bourne Shell is not available on SuSe - or did you start to support "obosh" binaries recently?
It is available for installation since April 2018 or so, but that is about it. I have not yet tried having it as /bin/sh. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2019-02-08 at 13:22 +0100, Martin Wilck wrote:
4. /bin/sh has pointed to bash for a long time (not sure how long exactly).
I think that SuSE 5.3 (1998) had the link, but I can verify later if it is of interest. - -- Cheers, Carlos E. R. (from openSUSE 15.0 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXF2IPRwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVbaoAnArqrP9flhHh9+KhvjCu QeNgf+0SAJ0fUAPAIaGp8X+kNqVt/Y/sb7cXYQ== =9JFg -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2019-02-08 at 14:25 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 13:22, Martin Wilck wrote:
I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it is not.
But it could be. In fact, it can be. `zypper dup` yesterday with an update block of 120 packages yielded only 7 scriptlet problems when sh is dash.
For me the question is, what's the benefit? I mean, anyone can choose whatever shell (s)he/ likes, but what's the point in having "/bin/sh" point to it? Do you uninstall bash on your systems? Is it size? Agreed, we could save ~800kiB in a minimal installation if we were able to use dash instead of bash as the OS's basic shell. But that would require a lot more work than just checking the scriptlets. Nowadays, bash is even in the initrd.
If we can't assume that /bin/sh is bash, what else can we assume? I recall from earlier work that writing really 100% compatible shell code for all kinds of environments is really hard.
No one needs to write sh-compatible code. The code just needs to be annotated that it requires a certain interpreter. Since script files already have a #! line most of the time, that's taken care of.
... unless that line reads '#! /bin/sh'.
That leaves script "snippets" that rely on implicit interpreters, such as in .spec.
Since I literally get to see every new line of spec code due to factory-review, many specfiles have already gotten their treatment (if one was even needed). The upside is that specfiles don't need a lot of work because everyone tries to keep them short anyway and reuse macros due to the volume of it all.
What rules do you follow in this review work? Do you require POSIX compliance? Or compliance with dash's features, which is not the same? Do other reviewers follow the same rules as you do? That's my point - IMO we'd need to define a reference and stick to it. I really don't care what it is, as long as it's POSIX or a superset of it. I've come to like bash's [[ ]] a lot but heck, I can live without it. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 8 Feb 2019 14:25:38 +0100 (CET), Jan Engelhardt <jengelh@inai.de> wrote:
No one needs to write sh-compatible code.
I do. But not on Linux My .sh scripts also have to run on ancient AIX and HP-UX, and neither comes with bash. Whenever I need more in a shell-script than old bourne syntax, I turn to perl. I /think/ I never wrote a sh/bash script that even uses $() -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
On Friday 2019-02-08 15:03, Martin Wilck wrote:
On Fri, 2019-02-08 at 14:25 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 13:22, Martin Wilck wrote:
I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it is not.
But it could be. In fact, it can be. `zypper dup` yesterday with an update block of 120 packages yielded only 7 scriptlet problems when sh is dash.
For me the question is, what's the benefit?
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
No one needs to write sh-compatible code. The code just needs to be annotated that it requires a certain interpreter. Since script files already have a #! line most of the time, that's taken care of.
... unless that line reads '#! /bin/sh'.
Debian is one distro where dash is prominently featured in one capacity or another, so upstream software already has to deal with the situation. Which leaves us with SUSE-specific software, and from a glance at e.g. github.com/openSUSE/sysconfig, they all quite consistently declare bash. I would argue that, in fact, people rather write #!/bin/bash these days just because it's in more or less every howto, and no newbie wants to have to deal the significant changes that a POSIX sh would bring.
Since I literally get to see every new line of spec code due to factory-review, many specfiles have already gotten their treatment (if one was even needed). The upside is that specfiles don't need a lot of work because everyone tries to keep them short anyway and reuse macros due to the volume of it all.
What rules do you follow in this review work?
* Does it build? No => back to the drawing board (reject) * https://en.opensuse.org/openSUSE:Creating_a_changes_file_(RPM) fulfilled? No => give up, complain (comment), or let someone else have the honor to reject it * https://en.opensuse.org/openSUSE:Package_description_guidelines fulfilled? No => complain or outright make an independent SR to the develprj/DP for evaluation and debate * https://en.opensuse.org/openSUSE:Packaging_guidelines#Licensing https://en.opensuse.org/openSUSE:Packaging_guidelines#Legal (only sometimes and only cursory, since there's a dedicated legal department and robots doing that already) metadata mismatch => reject * https://en.opensuse.org/openSUSE:Trademark_guidelines failure to abide => SR to DP (the bots don't catch it ATM) * https://en.opensuse.org/openSUSE:Packaging_guidelines and https://en.opensuse.org/openSUSE:Specfile_guidelines nonconformance => SR to DP, so I get the credit for fixing it * Anything else appears worth fixing => SR to DP
Do you require POSIX compliance?
No.
Or compliance with dash's features, which is not the same?
No.
Do other reviewers follow the same rules as you do?
Reviewers may forget some points sometimes letting bad stuff pass through, but they be forgiven for the guidelines are so plentiful and humans have a certain MTBF. So... yes, they follow the "rules". Do they follow the same algorithm? I suspect not everyone will jump in and proactively fix a submission for the submitter, because that constitutes a development not a review activity. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 8 Feb 2019 at 13:23, Martin Wilck <mwilck@suse.com> wrote:
Hi Jan, all,
I'd like to carry our OBS dicussion to a wider audience (https://build.opensuse.org/request/show/672510).
The question is whether we can assume that "/bin/sh" links to bash, in particular whether rpm scriptlets without explicit -p interpreter can be assumed to interpreted by bash.
I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it is not. Unless it's tampered with, /bin/sh is a symlink to /bin/bash. bash is not started in full POSIX mode if invoked as /bin/sh, and even if it's in POSIX mode, it supports some extensions over the POSIX shell spec (e.g. the [[ ]] construct), which makes it behave differently than another shell not supporting [[ ]] would (*). Problably there are more differences, I can't claim to know them all.
Here are some arguments why I think it'd be reasonable to assume that /bin/sh is bash on openSUSE:
1. patterns-base-minimal_base depends on bash, and the /bin/sh symlink is a non-configurable part of the "bash" package. 2. we could handle /bin/sh via /etc/alternatives, but we don't. 3. our Wiki suggests testing failing scriplets using "bash -xv" (https://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets) 4. /bin/sh has pointed to bash for a long time (not sure how long exactly).
FTR, Fedora basically guarantees (sh == bash) for the purpose of rpm scriptlets (https://fedoraproject.org/wiki/Packaging:Scriptlets). So Fedora <-> openSUSE portability may also be an issue to consider.
If we can't assume that /bin/sh is bash, what else can we assume? I recall from earlier work that writing really 100% compatible shell code for all kinds of environments is really hard. E.g., "[" isn't 100% portable either, even though it's part of the POSIX "test" standard (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html).
We should have clear rules which syntax expressions can be used in rpm scriptlets, and which can't. IMO we should define one of the various existing shells as a reference by saying "if it's supported by this shell, it's valid scriptlet code". That'd be easier (especially for testing compliance) than referring to a spec. That reference shell doesn't have to be bash, of course.
Thanks Martin
Thanks Martin, I see two questions popping out of this post and the subsequent discussion. I'm replying to the original because I wish to summarise my views from the thread to date, rather than respond to details specifically. I see Question 1) as "should packaging submissions be rejected because they assume /bin/sh is bash?" My answer to this is a strong resounding, maybe even slightly grumpy, "No" /bin/sh is currently not user configurable. There is no way, without removing files owned by an openSUSE distribution package, to set /bin/sh to be something other than /bin/bash. In a growing number of supported openSUSE configurations (Kubic, Transactional Server, etc) the user can't even do that given / is read-only. So, while I can see the benefit of avoiding bashisms, I really, really, strongly object to any SR getting held up in any way manner or form by any reviewer on the grounds that it assumes /bin/sh is pointing to /bin/bash. (Note, this really strong opinion is despite the fact that I'm a zsh fanboy who sets /bin/zsh to his login shell for every user I use, including root) However this debate does bring up a second question I see Question 2) as "should /bin/bash be /bin/sh in openSUSE?" I do not have a strong answer to this at the moment. I have never been comfortable with /bin/bash being /bin/sh, because of the quirky nature of 'bashisms'. But on the other hand, is there a suitably generic 'lowest common denominator' alternative? If the answer is yes, I can see how moving to something else could have benefits. For example, it has been mentioned dash would save ~800KiB or so compared to bash When you consider things like our container images, which don't really contain much more than zypper and bash, that 800KiB saved in every-single-container could be a huge benefit. Maybe there is a better option that is even smaller and more standards compliant? I'd still advocate for /bin/bash being the default shell for users (just like right now..we explicitly define /bin/bash in /etc/passwd, not /bin/sh) But I think /bin/sh should be the bare minimum, smallest, leanest, most generic shell. It needs to be something which any developer/packager should be sure will act the same way in every-single openSUSE system. This behaviour needs to be the same regardless of it being a server, desktop, container, JeOS image, or whatever, and should be confident that it will work in a very standards compliant way. I'm not sure it needs to be bash, I'm open to the idea of it changing, and I'm even willing to help with the insane amount of fallout that could be caused if we do decide to change it ;) Thoughts? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 15:03, Martin Wilck wrote:
For me the question is, what's the benefit?
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
That's truly interesting. But I can bounce back your argument: you could surely run "pbosh ./configure" to the same effect (what's pbosh, btw? did you mean posh?). Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins. There is some evidence that bash is not the fastest of shells ( https://unix.stackexchange.com/questions/148035/is-dash-or-some-other-shell-... ). OTOH, shell script performance matters much less these days than it used to.
Which leaves us with SUSE-specific software, and from a glance at e.g. github.com/openSUSE/sysconfig, they all quite consistently declare bash. I would argue that, in fact, people rather write #!/bin/bash these days just because it's in more or less every howto, and no newbie wants to have to deal the significant changes that a POSIX sh would bring.
But isn't that also an argument for having /bin/sh always point to bash? Like it or not, bash is the "de-facto standard" (yuck) shell today under Linux. People expect it to be that way.
Since I literally get to see every new line of spec code due to factory-review, many specfiles have already gotten their treatment (if one was even needed). The upside is that specfiles don't need a lot of work because everyone tries to keep them short anyway and reuse macros due to the volume of it all.
What rules do you follow in this review work?
* Does it build? No => back to the drawing board (reject) [...]
Thanks a lot for the extensive description, I was merely interested in your rules wrt to shell scriptlets :-)
* https://en.opensuse.org/openSUSE:Packaging_guidelines and https://en.opensuse.org/openSUSE:Specfile_guidelines nonconformance => SR to DP, so I get the credit for fixing it
* Anything else appears worth fixing => SR to DP
Do you require POSIX compliance?
No.
Or compliance with dash's features, which is not the same?
No.
I'm surprised - "No" to both of these, yet you reject bash-isms? Back to my point - openSUSE needs to have a reliable standard what kind of shell syntax is acceptable in scriptlets which don't use "-p". IOW, packagers need good guidance how to write scriptlets, and ideally a tool to verify that they comply. What should that standard be, and how would we define it, if not by "it complies iff it runs correctly under xyzsh" ? I'd be fine with xyzsh being dash rather than bash, but we need to settle down on _something_. Thanks, Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2019-02-08 at 18:10 +0100, Richard Brown wrote:
But I think /bin/sh should be the bare minimum, smallest, leanest, most generic shell. It needs to be something which any developer/packager should be sure will act the same way in every-single openSUSE system. This behaviour needs to be the same regardless of it being a server, desktop, container, JeOS image, or whatever, and should be confident that it will work in a very standards compliant way.
I'm not sure it needs to be bash, I'm open to the idea of it changing, and I'm even willing to help with the insane amount of fallout that could be caused if we do decide to change it ;)
You seem to be arguing for dash: https://wiki.ubuntu.com/DashAsBinSh. But that article also painfully demonstrates all the stuff people need to avoid if they want to write compliant code. Regards, Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 08.02.19 um 18:30 schrieb Martin Wilck:
You seem to be arguing for dash: https://wiki.ubuntu.com/DashAsBinSh. But that article also painfully demonstrates all the stuff people need to avoid if they want to write compliant code.
If we really want a small, fast, good shell, I'd vote for busybox's ash. It supports many useful bashisms without being obnoxious and annoying like that abomination that is dash. It's even smaller: seife@strolchi:/local/seife/src/git-repos/busybox> ls -lh _install/bin/ total 100K lrwxrwxrwx 1 seife users 7 Feb 8 19:02 ash -> busybox -rwxr-xr-x 1 seife users 100K Feb 8 19:02 busybox lrwxrwxrwx 1 seife users 7 Feb 8 19:02 sh -> busybox seife@strolchi:/local/seife/src/git-repos/busybox> ls -lh /usr/bin/dash -rwxr-xr-x 1 root root 124K Jan 30 02:39 /usr/bin/dash IMVHO dash serves only one purpose - as a bad example ;-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Martin Wilck <mwilck@suse.com> wrote:
On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 15:03, Martin Wilck wrote:
For me the question is, what's the benefit?
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
That's truly interesting. But I can bounce back your argument: you could surely run "pbosh ./configure" to the same effect (what's pbosh, btw? did you mean posh?).
"posh" is not a shell but a joke. It is full of bugs and cannot be seen as a serious POSIX shell implementation. "pbosh" is a compile variant of "bosh" which is the maintained portable and enhanced Bourne Shell that allows to be compiled compatible to the SVr4 Bourne Shell (obosh), a minimal strict POSIX variant (pbosh) and an enhanced POSIX variant (bosh). People who like to run "configure" tests as benchmarks should know that there is a need to make sure that sub-shells are called correctly. You need to call something like: CONFIG_SHELL=$shell $shell ./configure
Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.
This is not unfair. If you like to compare, I recommend you to use a ksh93 from 2008 which is the fastest shell ever and edit "configure" to switch off the important related ksh93 builtins. Note that ksh93 from 2008 is nearly twice as fast as bash. The most called programs are: rm cat sed tr mv in the order of usage. The reason why ksh93 is so fast is: - it uses vfork(), this is also true for bosh since 4 years. - it has a rewritten parser/interpreter that is larger (more code) but faster than the historic Bourne Shell This also applies to bosh - ksh93 does not fork for "(subcmd)". It rather creates a new set of "global variables" in a structure. This is ksh93 specific. - ksh93 uses an own i18n implementation with interface functions that better fit to the needs of a shell.
There is some evidence that bash is not the fastest of shells ( https://unix.stackexchange.com/questions/148035/is-dash-or-some-other-shell-... ). OTOH, shell script performance matters much less these days than it used to.
dash is approx. 30% faster than bash because it does not suport multi-byte characters.
But isn't that also an argument for having /bin/sh always point to bash? Like it or not, bash is the "de-facto standard" (yuck) shell today under Linux. People expect it to be that way.
BTW: Most of the bash "extensions" are taken from ksh93. The POSIX standard shell definition was derived from the features from ksh88 with [[ ]], (( )) and array variables removed. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Am 08.02.19 um 18:30 schrieb Martin Wilck:
You seem to be arguing for dash: https://wiki.ubuntu.com/DashAsBinSh. But that article also painfully demonstrates all the stuff people need to avoid if they want to write compliant code.
If we really want a small, fast, good shell, I'd vote for busybox's ash. It supports many useful bashisms without being obnoxious and annoying like that abomination that is dash.
ash based shells have many pitfalls. One important is that they do not implement $(cmd) correctly in corner cases. Also: the original ash (and dash) do not implement the required support for multi byte characters. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 2019-02-08 18:22, Martin Wilck wrote:
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
That's truly interesting. But I can bounce back your argument: you could surely run "pbosh ./configure" to the same effect (what's pbosh, btw? did you mean posh?).
But who knows if some command within all the code calls "sh", and then would get bash implicitly again. """bosh is a Bourne-style POSIX command interpreter (shell), similar to dash, but supports multibyte input/output."""
Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.
So, have bash invest a bit in optimizing the base dialect. It seems ridiculous that a mere syntactical difference leads to such great discrepancies: » for ((i=0; i<1000000;++i)); do echo '[ "$a$b" = 5 ]'; done >ca1 » for ((i=0; i<1000000;++i)); do echo '[[ "$a$b" = 5 ]]'; done >ca2 » time pbosh ca1 real 0m3.198s user 0m3.145s sys 0m0.052s » time bash ca1 real 0m3.987s user 0m3.985s sys 0m0.000s » time bash ca2 real 0m2.669s user 0m2.668s sys 0m0.000s If anything else, this reeks of embrace-extend-extinguish. First make [ suck, add [[, then make everyone write [[ just to be able to reach or beat the [ implementation of another sh. Something is wrong here.
OTOH, shell script performance matters much less these days than it used to.
The question is only to whom, and will point to https://xkcd.com/303/ .
Which leaves us with SUSE-specific software, and from a glance at e.g. github.com/openSUSE/sysconfig, they all quite consistently declare bash. I would argue that, in fact, people rather write #!/bin/bash these days just because it's in more or less every howto, and no newbie wants to have to deal the significant changes that a POSIX sh would bring.
But isn't that also an argument for having /bin/sh always point to bash? Like it or not, bash is the "de-facto standard" (yuck) shell today under Linux. People expect it to be that way.
"We always did it that way" is just not a very good argument for anything. I have not argued for changing the openSUSE defaults (not yet anyway). But I would like to have the possibility to switch everything that does not hardrequire bash to just use another sh on my system that may just be that tad bit faster that, in the spirit of xkcd303, improves my productivity at the end of the day. Think of what clang brought to the compiler ecosystem by being a mostly-drop-in replacement for gcc.
Do you require POSIX compliance?
No.
Or compliance with dash's features, which is not the same?
No.
I'm surprised - "No" to both of these, yet you reject bash-isms?
I do not reject submissions to openSUSE:Factory if they have bash-specific code. But I will make the proposal to either change them to sh, or annotate the section as "/bin/bash" (depending on triviality). If all a scriptlet does is [[ "$1" = 0 ]], then it's going to be [ "$1" = 0 ]. If however it tries funny things like ${x/a/b} or =~, it will just get annotated as #!/bin/bash.
Back to my point - openSUSE needs to have a reliable standard what kind of shell syntax is acceptable in scriptlets which don't use "-p".
The aim surely is sh, simply because most specfiles are simple, already compliant or annotated -- judging from all the files that have passed me by within the last two-plus years. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2019-02-08 at 20:01 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 18:22, Martin Wilck wrote:
But isn't that also an argument for having /bin/sh always point to bash? Like it or not, bash is the "de-facto standard" (yuck) shell today under Linux. People expect it to be that way.
"We always did it that way" is just not a very good argument for anything.
That wasn't my argument. The argument was that changing it would break people's assumptions. But never mind.
Back to my point - openSUSE needs to have a reliable standard what kind of shell syntax is acceptable in scriptlets which don't use "- p".
The aim surely is sh, simply because most specfiles are simple, already compliant or annotated -- judging from all the files that have passed me by within the last two-plus years.
Define "sh", please. We've seem at least 4 candidates in this thread so far except bash itself (dash, ash, (p)bosh, ksh93). Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2019-02-08 at 19:07 +0100, Joerg Schilling wrote:
Martin Wilck <mwilck@suse.com> wrote:
On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 15:03, Martin Wilck wrote:
For me the question is, what's the benefit?
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
People who like to run "configure" tests as benchmarks should know that there is a need to make sure that sub-shells are called correctly.
You need to call something like:
CONFIG_SHELL=$shell $shell ./configure
Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.
This is not unfair. If you like to compare, I recommend you to use a ksh93 from 2008 which is the fastest shell ever and edit "configure" to switch off the important related ksh93 builtins. Note that ksh93 from 2008 is nearly twice as fast as bash.
Speaking about shell script performance, IMO the only set of scripts for which this matters in practice is dracut, which would benefit a LOT from 30% speedup. Unfortunately it was written as a "#! /bin/bash" script in the first place. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Feb 11, 2019 at 10:21:47AM +0100, Martin Wilck wrote:
Speaking about shell script performance, IMO the only set of scripts for which this matters in practice is dracut, which would benefit a LOT from 30% speedup. Unfortunately it was written as a "#! /bin/bash" script in the first place.
Indeed, nevertheless as dracut uses bashism it should enforce to use /bin/bash, otherwise arrays, types, [[..]], ((..)), ... had been rewritten and maybe other external tools must be added as replacements. Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
On 11/02/2019 19:51, Martin Wilck wrote:
On Fri, 2019-02-08 at 19:07 +0100, Joerg Schilling wrote:
Martin Wilck <mwilck@suse.com> wrote:
On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 15:03, Martin Wilck wrote:
For me the question is, what's the benefit?
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
People who like to run "configure" tests as benchmarks should know that there is a need to make sure that sub-shells are called correctly.
You need to call something like:
CONFIG_SHELL=$shell $shell ./configure
Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.
This is not unfair. If you like to compare, I recommend you to use a ksh93 from 2008 which is the fastest shell ever and edit "configure" to switch off the important related ksh93 builtins. Note that ksh93 from 2008 is nearly twice as fast as bash.
Speaking about shell script performance, IMO the only set of scripts for which this matters in practice is dracut, which would benefit a LOT from 30% speedup. Unfortunately it was written as a "#! /bin/bash" script in the first place.
Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Mon, 11 Feb 2019 21:16:36 +1030 Simon Lees <sflees@suse.de> wrote:
On 11/02/2019 19:51, Martin Wilck wrote:
On Fri, 2019-02-08 at 19:07 +0100, Joerg Schilling wrote:
Martin Wilck <mwilck@suse.com> wrote:
On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:
On Friday 2019-02-08 15:03, Martin Wilck wrote:
For me the question is, what's the benefit?
Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.
People who like to run "configure" tests as benchmarks should know that there is a need to make sure that sub-shells are called correctly.
You need to call something like:
CONFIG_SHELL=$shell $shell ./configure
Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.
This is not unfair. If you like to compare, I recommend you to use a ksh93 from 2008 which is the fastest shell ever and edit "configure" to switch off the important related ksh93 builtins. Note that ksh93 from 2008 is nearly twice as fast as bash.
Speaking about shell script performance, IMO the only set of scripts for which this matters in practice is dracut, which would benefit a LOT from 30% speedup. Unfortunately it was written as a "#! /bin/bash" script in the first place.
Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.
And that is the question. Will it make any difference? Many packages that build slow spend a lot of time actually *compiling* stuff. Many packages that build slow spend a lot of time in rpm checkers. At least in one case it has been shown that rpm checkers would benefit from using all of bash features such as [[]] because it has much common functionality built-in and much time is spent forking off tools to do small tasks that a POSIX-compliant shell cannot do on its own but bash can. So which are the packages that actually do benefit from faster bare-bones /bin/sh? Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Engelhardt <jengelh@inai.de> wrote:
Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.
So, have bash invest a bit in optimizing the base dialect. It seems ridiculous that a mere syntactical difference leads to such great discrepancies:
» for ((i=0; i<1000000;++i)); do echo '[ "$a$b" = 5 ]'; done >ca1 » for ((i=0; i<1000000;++i)); do echo '[[ "$a$b" = 5 ]]'; done >ca2 » time pbosh ca1 real 0m3.198s user 0m3.145s sys 0m0.052s » time bash ca1 real 0m3.987s user 0m3.985s sys 0m0.000s » time bash ca2 real 0m2.669s user 0m2.668s sys 0m0.000s
If anything else, this reeks of embrace-extend-extinguish. First make [ suck, add [[, then make everyone write [[ just to be able to reach or beat the [ implementation of another sh. Something is wrong here.
Interesting catch! BTW: ksh93 has less differences between [ and [[. Also note that this is a ksh88 feature that has been introduced in order to avoid problems when the position of an operator may be a shell macro and this has been introduced before POSIX rearranged the test(1) definition to mainly depend on the number of parameters. PPS: the next version of bosh will be approx. 15% faster with this command and should be approx. as fast as bash with [[ now. I just make a minor change to the memory management module. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 2019-02-11 12:07, Michal Suchánek wrote:
Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.
Many packages that build slow spend a lot of time actually *compiling* stuff.
Yes and no. Build times follow the usual long-tail distribution (Pareto or something). Hugeass packages like chromium of course spent 99% their time in g++, while things like python-* are probably spending most of their time in /usr/bin/build and /usr/bin/rpm and disk IO. And we don't exactly have few packages in the {python, perl, ruby} camp, mind you.
Many packages that build slow spend a lot of time in rpm checkers.
But for different reasons. * The compression time scales linearly with the size of the contents. * absolute executable size roughly scales linearly with package size (something in the ballpark of 60% of a package are executables normally; ignoring noarch packages) * ELF analysis (dwz et al) scales linearly with executable size * the size of executable scales linearly with project size Which is the reason things like chromium require exorbitant time and are very frustrating if it happens to fail at the last moment, while packages like cdecl are comparatively trivial - but have a high buildroot setup cost. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Michal Suchánek <msuchanek@suse.de> wrote:
Many packages that build slow spend a lot of time actually *compiling* stuff.
Are you shure? Before I did run performance analysis on "configure", I thought that the compiler is the most time consuming thing in this area and I failed completely with this assumption. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, 11 Feb 2019 12:18:58 +0100 (CET) Jan Engelhardt <jengelh@inai.de> wrote:
On Monday 2019-02-11 12:07, Michal Suchánek wrote:
Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.
Many packages that build slow spend a lot of time actually *compiling* stuff.
Yes and no. Build times follow the usual long-tail distribution (Pareto or something). Hugeass packages like chromium of course spent 99% their time in g++, while things like python-* are probably spending most of their time in /usr/bin/build and /usr/bin/rpm and disk IO. And we don't exactly have few packages in the {python, perl, ruby} camp, mind you.
Many packages that build slow spend a lot of time in rpm checkers.
But for different reasons. * The compression time scales linearly with the size of the contents. * absolute executable size roughly scales linearly with package size (something in the ballpark of 60% of a package are executables normally; ignoring noarch packages) * ELF analysis (dwz et al) scales linearly with executable size * the size of executable scales linearly with project size
Which is the reason things like chromium require exorbitant time and are very frustrating if it happens to fail at the last moment, while packages like cdecl are comparatively trivial - but have a high buildroot setup cost.
And you forgot to quote the important part: is the setup cost mitigated by shell scripts that use fast bare-bones shell and have to fork() for every tiny task or shell scripts that fully use bash builtins? Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Michal Suchánek <msuchanek@suse.de> wrote:
And you forgot to quote the important part: is the setup cost mitigated by shell scripts that use fast bare-bones shell and have to fork() for every tiny task or shell scripts that fully use bash builtins?
Huge shells that call fork() instead of vfork() are typically slow. ksh93 and bosh use vfork() whereever possible. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, 2019-02-11 at 12:24 +0100, Joerg Schilling wrote:
Michal Suchánek <msuchanek@suse.de> wrote:
Many packages that build slow spend a lot of time actually *compiling* stuff.
Are you shure?
Before I did run performance analysis on "configure", I thought that the compiler is the most time consuming thing in this area and I failed completely with this assumption.
IIRC even "configure" spends a lot of time compiling, as it forks the compiler many times for testing things. Indeed there are packages where "configure" takes substantially more time that the actual "make". Anyway, for speeding up "configure", we could simply replace the "./configure" call with "$CONFIG_SHELL=$someshell $someshell ./configure" in those spec files where this really matters. So I reckon that "configure" speed is not a strong argument for making /bin/sh customizable. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, 11 Feb 2019 12:40:32 +0100 Martin Wilck <mwilck@suse.com> wrote:
On Mon, 2019-02-11 at 12:24 +0100, Joerg Schilling wrote:
Michal Suchánek <msuchanek@suse.de> wrote:
Many packages that build slow spend a lot of time actually *compiling* stuff.
Are you shure?
Before I did run performance analysis on "configure", I thought that the compiler is the most time consuming thing in this area and I failed completely with this assumption.
IIRC even "configure" spends a lot of time compiling, as it forks the compiler many times for testing things. Indeed there are packages where "configure" takes substantially more time that the actual "make".
Anyway, for speeding up "configure", we could simply replace the "./configure" call with
"$CONFIG_SHELL=$someshell $someshell ./configure"
And even have %configure macro do it for you. Thanks Michal
in those spec files where this really matters. So I reckon that "configure" speed is not a strong argument for making /bin/sh customizable.
Martin
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 11/02/2019 21:48, Jan Engelhardt wrote:
On Monday 2019-02-11 12:07, Michal Suchánek wrote:
Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.
Many packages that build slow spend a lot of time actually *compiling* stuff.
Yes and no. Build times follow the usual long-tail distribution (Pareto or something). Hugeass packages like chromium of course spent 99% their time in g++, while things like python-* are probably spending most of their time in /usr/bin/build and /usr/bin/rpm and disk IO. And we don't exactly have few packages in the {python, perl, ruby} camp, mind you.
Many packages that build slow spend a lot of time in rpm checkers.
But for different reasons. * The compression time scales linearly with the size of the contents. * absolute executable size roughly scales linearly with package size (something in the ballpark of 60% of a package are executables normally; ignoring noarch packages) * ELF analysis (dwz et al) scales linearly with executable size * the size of executable scales linearly with project size
Yeah for many larger packages up to 1/3rd of the build time is spent after make install is finished, but for larger slower building packages as mentioned above the issue is not the rpm checkers, for slow, large packages the rpm checkers only make up a very small amount of the build time whereas for small quick packages the rpm checkers take a larger percentage of time which is why we already invested a bit of time in speeding them up. For large packages the time is mostly spent doing compression and extracting the debug symbols, this is currently not done in parallel so we waste huge amounts of CPU time here currently. Performing this step in parallel by processing multiple subpackages at once does provide a significant improvement, however given rpmbuild was never designed to do this in parallel I am yet to be able to do it reliably without crashes due to race conditions accessing memory or deadlocks so i'm not actively working on it atm. Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium. If we can get to a point where we are speeding up these kind of packages by some amount maybe it makes sense to add an update alternatives setup for #!/bin/sh even if its mostly so we can use it in the build root of certain packages to speed them up. At the moment our team is still looking into a bunch of smaller ways to speed up the build in the hope if we find enough it will eventually get to something more substantial, its an area people haven't spent a lot of time looking at so we are still finding some low hanging fruit. From our point of view we are trying to reduce the time it takes to release security updates and one of the ways to do that is obviously speeding up build times. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Tuesday, 12 February 2019 1:59 Simon Lees wrote:
Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium.
It certainly can. Few years ago, I was rather frustrated that when building a kernel package locally, more than 6 minutes out of total 35 was spent by brp-symlinks script. I was even considering to rewrite it to C and parallelize as a hackweek project. Then Tomáš Čech tried, as an exercise, to rewrite the script just by replacing various invokations of sed, grep etc. with bash specific expansions. As a result, he managed to squeeze those 6 minutes down to ~30 seconds. Michal Kubeček -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Feb 12, 2019 at 07:09:18AM +0100, Michal Kubecek wrote:
On Tuesday, 12 February 2019 1:59 Simon Lees wrote:
Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium.
It certainly can. Few years ago, I was rather frustrated that when building a kernel package locally, more than 6 minutes out of total 35 was spent by brp-symlinks script. I was even considering to rewrite it to C and parallelize as a hackweek project.
Then Tomáš Čech tried, as an exercise, to rewrite the script just by replacing various invokations of sed, grep etc. with bash specific expansions. As a result, he managed to squeeze those 6 minutes down to ~30 seconds.
I can second this. Using bash builtins and string features can speed up scripts a lot. That is avoiding forking often for external command within loops, using <() fifo together with an external command to handle large lists of lines of strings at once and read the resulting lines with loops only using bash builtins. Also ksh93 loops can read from pipe (which is actual a combination of two socketpair()s) plus avoiding subprocess for the loop its self. But this is not portable to many bourne shell scripts as with ksh93 variables within the loop are visiable outside the loop. This can be an advantage but may break scripts depending on orignal bourne shell behaviour. The bash provides the <() fifo as a replacment for this. Then one can use the redirection operator to read from the <() fifo as stdin. -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
On Tue, 12 Feb 2019 08:34:00 +0100, "Dr. Werner Fink" <werner@suse.de> wrote:
On Tue, Feb 12, 2019 at 07:09:18AM +0100, Michal Kubecek wrote:
On Tuesday, 12 February 2019 1:59 Simon Lees wrote:
Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium.
It certainly can. Few years ago, I was rather frustrated that when building a kernel package locally, more than 6 minutes out of total 35 was spent by brp-symlinks script. I was even considering to rewrite it to C and parallelize as a hackweek project.
Then Tomáš Čech tried, as an exercise, to rewrite the script just by replacing various invokations of sed, grep etc. with bash specific expansions. As a result, he managed to squeeze those 6 minutes down to ~30 seconds.
I can second this. Using bash builtins and string features can speed up scripts a lot. That is avoiding forking often for external command within loops, using <() fifo together with an external command to handle large lists of lines of strings at once and read the resulting lines with loops only using bash builtins.
Also ksh93 loops can read from pipe (which is actual a combination of two socketpair()s) plus avoiding subprocess for the loop its self. But this is not portable to many bourne shell scripts as with ksh93 variables within the loop are visiable outside the loop. This can be an advantage but may break scripts depending on orignal bourne shell behaviour. The bash provides the <() fifo as a replacment for this. Then one can use the redirection operator to read from the <() fifo as stdin.
Maybe going into the area of being off-topic, but be careful to not being too enthusiastic here and do this for non-kernel distribution build packages (like LibreOffice or other open-source projects that live beyond Linux) Having developers optimize their build procedure to use the tools and features that are available in the "default" Linux world cause real big pain for OSS porters on different systems like HP-UX, AIX, Solaris etc Projects that require features - like bash functionality - that is not available on those proprietary OS's will cause the support on those OS's to halt. e.g. • GNU gcc is by now impossible to build on HP-UX. • Projects that switch(ed) to meson/ninja won't build outside Linux • Projects that require new(er/est) gcc won't build on OS's that won't build gcc I know that OSS developer time is "free" time and voluntary work, but remember that choosing the tool-of-the-day will cause headaches for those that use their free time to port OSS to those OS's that are not mainstream anymore. Story from the past: we had to switch from SCCS to something more recent. The team chose SVN. Our main development environment by the time was HP-UX 11.23, and we had to give up on SVN: we needed a week to build SVN from scratch with all its dependencies only to see it horribly fail. We then decided to try git. We just needed 3 hours to get it up and running, including the GUI's. *I* do not regret this, as I had a preference of git over svn from the start, and by now all our developers are used to git/gitlab and it is part of daily routine. If bash were to switch from make to meson/ninja, you'd rule out the use of modern bash on those weird OS's, as bash still is relative easy to build/port. (and so is "make") -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
On 12/02/2019 16:39, Michal Kubecek wrote:
On Tuesday, 12 February 2019 1:59 Simon Lees wrote:
Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium.
It certainly can. Few years ago, I was rather frustrated that when building a kernel package locally, more than 6 minutes out of total 35 was spent by brp-symlinks script. I was even considering to rewrite it to C and parallelize as a hackweek project.
Then Tomáš Čech tried, as an exercise, to rewrite the script just by replacing various invokations of sed, grep etc. with bash specific expansions. As a result, he managed to squeeze those 6 minutes down to ~30 seconds.
Michal Kubeček
Thats probably why I didn't notice it so badly now 30 seconds compared to the rest of the build time on some arch's is tiny. In the last year or two we modified the script to use GNU parallel so now most of the tests should also run in parallel as well, which generally means they take as long as the longest test, there are many packages that now only spend a second or two in these checks rather then 6-8 in the past. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Tue, 2019-02-12 at 11:29 +1030, Simon Lees wrote:
Yeah for many larger packages up to 1/3rd of the build time is spent after make install is finished,
Out of the checks we run, only "post-build-checks" consists of shell code. rpmlint and rpmlint-mini are python, and brp-check-suse is perl. And "post-build-checks" script use a "bash" shebang. The same holds for the "build" package. So IMO nothing would be gained wrt build checks by changing what /bin/sh points at. It'd make more sense to review "post-build-checks" and "build" for bashisms, and possibly use a different, faster shell for them. That'd be much less review work than doing it for the whole distro.
Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at.
As mentioned already, for that it'd be sufficient to use a different shell for running "configure" (iow modify the %configure macro). Regards Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
"Dr. Werner Fink" <werner@suse.de> wrote:
I can second this. Using bash builtins and string features can speed up scripts a lot. That is avoiding forking often for external command within loops, using <() fifo together with an external command to handle large lists of lines of strings at once and read the resulting lines with loops only using bash builtins.
In ksh93, <() works using /dev/fd/* The main difference to $(cmd) is that the output of the command is a normal pipe instead of letting the shell read from that pipe and create an argument list from it. For bash that did not yet rewrite it's parser to implement more efficient pipelines, this may give an advantage, but for shells that run the rightmost command in the main shell in case it is a builtin, there is no difference. So this does not give performance benefits in ksh93 and it would not give performance benefits if bosh did start to implement <().
Also ksh93 loops can read from pipe (which is actual a combination of two socketpair()s) plus avoiding subprocess for the loop its self. But this is not portable to many bourne shell scripts as with ksh93 variables within the loop are visiable outside the loop. This can be an advantage but may break scripts depending on orignal bourne shell behaviour. The bash provides the <() fifo as a replacment for this. Then one can use the redirection operator to read from the <() fifo as stdin.
See above. The original Bourne Shell implementation from 1976 had a main problem: program size could not exceed 64k on a PDP11 unless you used slow overlays (as used in vi). So the Bourne Shell has been written in a way that created the smallest program regardless of the speed of that program. ksh93 uses a new method to create pipelines. This is faster and it allows to use vfork() to speed up things. vfork() on a real UNIX system is typically 3-4x faster than fork() as it does not need to copy the address space description. This still applies even to modern UNIX versions like SunOS-4.x that introduced a copy on write fork() in 1988. This is where Linux could speed up BTW...as vfork() is using a copy on write based emulation on top of fork(). To be able to use vfork(), you need to use a different chain of processes when creating the processes for a pipeline. This requires a rewrite of the parser and the interpreter in the shell. Once you did this, it is not a big deal anymore to run the rightmost program of a pipeline in the main shell process in case it is a builtin command. This is what finally allows you to set up shell macros that are seen by the main shell, as in: echo bla | read VAR Another strange thing from the historic Bourne Shell is that a while loop with I/O redirection always has been run in a subshell. Ksh93 changed this. Since bosh also implements both optimizations mentioned above, typical shell scripts like "configure" now run 30% faster. ksh93 uses virtual sub-shells to avoid fork() in many cases and gets another speed up, but the version of ksh93 that is created by RedHat people on github is no longer faster than "bosh" as important code has been removed or destroyed. BTW: if you check the speed of "configure" in special, you will see that "echo" and "test" being a builtin makes the biggest difference and with GNU configure past 2.13, the biggest time consumer is the fact that "printf" is used instead of "echo" and if that is not built into the shell, a configure run on such a shell is extremely slow. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Feb 12, 2019 at 11:51:37AM +0100, Joerg Schilling wrote:
"Dr. Werner Fink" <werner@suse.de> wrote:
I can second this. Using bash builtins and string features can speed up scripts a lot. That is avoiding forking often for external command within loops, using <() fifo together with an external command to handle large lists of lines of strings at once and read the resulting lines with loops only using bash builtins.
In ksh93, <() works using /dev/fd/*
The main difference to $(cmd) is that the output of the command is a normal pipe instead of letting the shell read from that pipe and create an argument list from it.
As former maintainer of the ksh93 I'm aware, bash does it similar
For bash that did not yet rewrite it's parser to implement more efficient pipelines, this may give an advantage, but for shells that run the rightmost command in the main shell in case it is a builtin, there is no difference. So this does not give performance benefits in ksh93 and it would not give performance benefits if bosh did start to implement <().
Also ksh93 loops can read from pipe (which is actual a combination of two socketpair()s) plus avoiding subprocess for the loop its self. But this is not portable to many bourne shell scripts as with ksh93 variables within the loop are visiable outside the loop. This can be an advantage but may break scripts depending on orignal bourne shell behaviour. The bash provides the <() fifo as a replacment for this. Then one can use the redirection operator to read from the <() fifo as stdin.
See above. The original Bourne Shell implementation from 1976 had a main problem: program size could not exceed 64k on a PDP11 unless you used slow overlays (as used in vi). So the Bourne Shell has been written in a way that created the smallest program regardless of the speed of that program.
ksh93 uses a new method to create pipelines. This is faster and it allows to use vfork() to speed up things. vfork() on a real UNIX system is typically 3-4x faster than fork() as it does not need to copy the address space description. This still applies even to modern UNIX versions like SunOS-4.x that introduced a copy on write fork() in 1988. This is where Linux could speed up BTW...as vfork() is using a copy on write based emulation on top of fork().
To be able to use vfork(), you need to use a different chain of processes when creating the processes for a pipeline. This requires a rewrite of the parser and the interpreter in the shell. Once you did this, it is not a big deal anymore to run the rightmost program of a pipeline in the main shell process in case it is a builtin command. This is what finally allows you to set up shell macros that are seen by the main shell, as in:
echo bla | read VAR
Another strange thing from the historic Bourne Shell is that a while loop with I/O redirection always has been run in a subshell. Ksh93 changed this.
Since bosh also implements both optimizations mentioned above, typical shell scripts like "configure" now run 30% faster. ksh93 uses virtual sub-shells to avoid fork() in many cases and gets another speed up, but the version of ksh93 that is created by RedHat people on github is no longer faster than "bosh" as important code has been removed or destroyed.
BTW: if you check the speed of "configure" in special, you will see that "echo" and "test" being a builtin makes the biggest difference and with GNU configure past 2.13, the biggest time consumer is the fact that "printf" is used instead of "echo" and if that is not built into the shell, a configure run on such a shell is extremely slow.
Hmmm ... AFAICR ksh93 was huge collection of bugs causing a lot of bugzilla entries (tagged 56 bugs). I can remember the problems the $(cmd) had caused whereas the old `cmd` code had worked flawless. I had spent a lot of time to hunt bugs in the multibyte code as well. One bug was memory allocing code in the signal handling. During maintaining systemd I had no time anymore to hunt further bugs down hence the new maintainer for ksh93/libast. From my point of view I prefere a stable shell with maintainable code more than a exterm fast shell with bugs and interwoven bad documented code. -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
Projects that require features - like bash functionality - that is not available on those proprietary OS's will cause the support on those OS's to halt. e.g.
If you depend on proprietary bash features, you are not portable. This is why POSIX exists. I modified a few shell scripts from OpenSolaris to no longer depend on ksh93 to be able to use "bosh" as /bin/sh and /sbin/sh for the basic startup before /usr gets mounted. On the other side: as long as bash stays portable it is not a real problem to use bash as long as you put #!/bin/bash in the first line of your script. I guess this will not change as long as bash stays a one man show with the original maintainer.
? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc
It is bad to see that some GNU projects more and more implement the embrace and extent method from Microcoft to create a vendor lock in instead of using POSIX features that are portable.
I know that OSS developer time is "free" time and voluntary work, but remember that choosing the tool-of-the-day will cause headaches for those that use their free time to port OSS to those OS's that are not mainstream anymore.
This is e.g. why the ksh93 from Redhat no longer compiles on Solaris. The background in this case BTW. is missing skills by the related people.
Story from the past: we had to switch from SCCS to something more recent. The team chose SVN. Our main development environment by the time was HP-UX 11.23, and we had to give up on SVN: we needed a week to build SVN from scratch with all its dependencies only to see it horribly fail. We then decided to try git. We just needed 3 hours to get it up and running, including the GUI's. *I* do not regret this, as I had a preference of git over svn from the start, and by now all our developers are used to git/gitlab and it is part of daily routine.
Well, I could never understand why peole would like to use svn as it is as slow as CVS and as it did even cause some projects to loose all their history after a internal change of the datastructures.
If bash were to switch from make to meson/ninja, you'd rule out the use of modern bash on those weird OS's, as bash still is relative easy to build/port. (and so is "make")
Well writing makefiles that depend on "gmake" is a bad practice, but it is still better than using something else. If you are interested in a portable POSIX certified make program, I recommend to use the "SunPro Make" from schilytools. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
"Dr. Werner Fink" <werner@suse.de> wrote:
Hmmm ... AFAICR ksh93 was huge collection of bugs causing a lot of bugzilla entries (tagged 56 bugs). I can remember the problems the $(cmd) had caused whereas the old `cmd` code had worked flawless. I had spent a lot of time to hunt bugs in the multibyte code as well. One bug was memory allocing code in the signal handling. During maintaining systemd I had no time anymore to hunt further bugs down hence the new maintainer for ksh93/libast. From my point of view I prefere a stable shell with maintainable code more than a exterm fast shell with bugs and interwoven bad documented code.
Such problems did not exist on Solaris and a non-portable ksh93 kills it. The main advantage from ksh93 was is portability as long as David Korn was the maintainer. I don't like the current version of ksh93 even though it fixed a few bugs. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 12/02/2019 20:09, Martin Wilck wrote:
On Tue, 2019-02-12 at 11:29 +1030, Simon Lees wrote:
Yeah for many larger packages up to 1/3rd of the build time is spent after make install is finished,
Out of the checks we run, only "post-build-checks" consists of shell code. rpmlint and rpmlint-mini are python, and brp-check-suse is perl. And "post-build-checks" script use a "bash" shebang. The same holds for the "build" package.
So IMO nothing would be gained wrt build checks by changing what /bin/sh points at. It'd make more sense to review "post-build-checks" and "build" for bashisms, and possibly use a different, faster shell for them. That'd be much less review work than doing it for the whole distro.
Well yeah as I was trying to say while pointing out we spend a significant time post compilation, so its not just all compile time, we also don't spend a significant time in these checks. Its all spent in RPM so no i'm not expecting that we'd speed up these checks by changing shell, we don't spend enough time running these checks atm for it to be worth investigating further here unless the stuff is really obvious. For configure / make / make install there is still more we can investigate to see if its worth it then we can look at the various options if its actually worth going forward whether thats different macros or a different #!/bin/sh -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On 12/02/2019 21:50, Joerg Schilling wrote:
H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
Projects that require features - like bash functionality - that is not available on those proprietary OS's will cause the support on those OS's to halt. e.g.
If you depend on proprietary bash features, you are not portable. This is why POSIX exists. I modified a few shell scripts from OpenSolaris to no longer depend on ksh93 to be able to use "bosh" as /bin/sh and /sbin/sh for the basic startup before /usr gets mounted.
On the other side: as long as bash stays portable it is not a real problem to use bash as long as you put #!/bin/bash in the first line of your script. I guess this will not change as long as bash stays a one man show with the original maintainer.
? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change. https://twitter.com/mesonbuild/status/1095391825647538176 -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Wed, 2019-02-13 at 09:50 +1030, Simon Lees wrote:
On 12/02/2019 21:50, Joerg Schilling wrote:
H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
Projects that require features - like bash functionality - that is not available on those proprietary OS's will cause the support on those OS's to halt. e.g.
If you depend on proprietary bash features, you are not portable. This is why POSIX exists. I modified a few shell scripts from OpenSolaris to no longer depend on ksh93 to be able to use "bosh" as /bin/sh and /sbin/sh for the basic startup before /usr gets mounted.
On the other side: as long as bash stays portable it is not a real problem to use bash as long as you put #!/bin/bash in the first line of your script. I guess this will not change as long as bash stays a one man show with the original maintainer.
? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.
Meson being a python3 app would be a terrible choice for anythingn in the distro bootstrap cycle (aka Ring0). That cycle is large enough as it is - pulling python3 into it won''t make it better. So I hope the idea of bash moving to meson is just a random though nobody really considers. Cheers Dominique
On Wednesday, 13 February 2019 0:20 Simon Lees wrote:
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.
It's hard to believe just switching to meson could achieve this without something being deeply wrong with the previous build system and that one couldn't improve the build times significantly even without meson. Michal Kubecek -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 13 Feb 2019 09:50:51 +1030, Simon Lees <sflees@suse.de> wrote:
? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.
Maybe they forgot that make supports -j35 meson is "parallel" by default, make is not And I fully subscribe to Dominique's comment, but that might be obvious -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
On Feb 13 2019, Dominique Leuenberger / DimStar <dimstar@opensuse.org> wrote:
Meson being a python3 app would be a terrible choice for anythingn in the distro bootstrap cycle (aka Ring0). That cycle is large enough as it is - pulling python3 into it won''t make it better.
You might be surprised that it's already present. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Feb 13 2019, Simon Lees <sflees@suse.de> wrote:
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.
That only tells how terrible the ksh build system was. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Simon Lees <sflees@suse.de> wrote:
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.
This could be seen as fake news. If you remove more than 95% of the code, the remaining rest of course compiles faster. A more impressive speedup could have been created by e.g. pre-compiling "nmake" once and installing the result before starting to build the code. Switching to Meson made KornShell non-portable :-( Removing the malloc() implementation from libast and the i18n implementation that is also from libast made ksh93 noticeable slower. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Michal Kubecek <mkubecek@suse.cz> wrote:
On Wednesday, 13 February 2019 0:20 Simon Lees wrote:
Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.
It's hard to believe just switching to meson could achieve this without something being deeply wrong with the previous build system and that one couldn't improve the build times significantly even without meson.
It is of course not true. The speedup is a result of only including a reduced ksh93 instead of compiling the **whole AST suite** that even included the AT&T ANSI CPP in order to be able to compile the suite using the Microsoft C compiler with POSIX pathnames. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andreas Schwab <schwab@suse.de> wrote:
That only tells how terrible the ksh build system was.
It was portable, so it was definitely better that what these people introcuded. So in general, be careful about fake news.... Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 2019-02-08 18:10, Richard Brown wrote:
On Fri, 8 Feb 2019 at 13:23, Martin Wilck <mwilck@suse.com> wrote:
I'd like to carry our OBS dicussion to a wider audience (https://build.opensuse.org/request/show/672510).
The question is whether we can assume that "/bin/sh" links to bash,
[...] But I think /bin/sh should be the bare minimum, smallest, leanest, most generic shell. I'm not sure it needs to be bash, I'm open to the idea of it changing, and I'm even willing to help with the insane amount of fallout that could be caused if we do decide to change it ;)
Thoughts?
Perhaps you can help your colleague markkp on SR 676611, he seems to have a severe difficulty in understanding the submission, particularly the sh part. See, I am just trying to reduce the aforementioned fallout before it happens. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi all, On Mon, Feb 18, 2019 at 9:41 AM Jan Engelhardt <jengelh@inai.de> wrote:
On Friday 2019-02-08 18:10, Richard Brown wrote:
On Fri, 8 Feb 2019 at 13:23, Martin Wilck <mwilck@suse.com> wrote:
I'd like to carry our OBS dicussion to a wider audience (https://build.opensuse.org/request/show/672510).
The question is whether we can assume that "/bin/sh" links to bash,
[...] But I think /bin/sh should be the bare minimum, smallest, leanest, most generic shell. I'm not sure it needs to be bash, I'm open to the idea of it changing, and I'm even willing to help with the insane amount of fallout that could be caused if we do decide to change it ;)
Thoughts?
Perhaps you can help your colleague markkp on SR 676611, he seems to have a severe difficulty in understanding the submission, particularly the sh part. See, I am just trying to reduce the aforementioned fallout before it happens.
I might have missed part of this (very long) discussion, but in my opinion - using /bin/sh means you SHOULD only use constructions which are POSIX compliant. - when you want to use bashisms, use /bin/bash. The same applies to other shells. I don't see why you ever want to deviate from that? Sticking to these two rules keeps code portable, readable and actually more secure as you know what to expect: bashisms might work differently in other shells. Also note that bash behaves slightly different when invokes as 'sh' (see man page)... If I read it correctly, these rules is what Jan tried to implement in https://build.opensuse.org/request/show/676611 which I would fully support! Mischa -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Mischa Salle <mischa.salle@gmail.com> wrote:
I might have missed part of this (very long) discussion, but in my opinion - using /bin/sh means you SHOULD only use constructions which are POSIX compliant. - when you want to use bashisms, use /bin/bash. The same applies to other shells.
This is finally the right conclusion even thouh POSIX does not make any statements on what /bin/sh is. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, 2019-02-18 at 13:36 +0100, Joerg Schilling wrote:
Mischa Salle <mischa.salle@gmail.com> wrote:
I might have missed part of this (very long) discussion, but in my opinion - using /bin/sh means you SHOULD only use constructions which are POSIX compliant. - when you want to use bashisms, use /bin/bash. The same applies to other shells.
This is finally the right conclusion even thouh POSIX does not make any statements on what /bin/sh is.
I don't think that this is the right conclusion. Here's why. I've been told in this discussion that the POSIX shell standard for "allows extensions". IOW, it only defines a minimal set of features, and implementations are free to add more features on top and still "comply" with the standard. Therefore the fact that bash, invoked as "/bin/sh", supports the non-POSIX '[[ ]]' syntax doesn't make it non- compliant. We need to define without ambiguity under what conditions a _shell script_ complies with the feature set of "/bin/sh". That means that we need to have a _test procedure_ (#). We could say that "code is compliant iff it runs without errors under every POSIX-compliant shell (*)". But that's hard to test, as we'd have to test with every POSIX- compliant shell out there, and we may not even know them all. We also don't want to test by human beings parsing the code (even if that's basically what Jan is doing today). Rather, we need to settle on an actual shell implementation, and define "compliant" by reference to this implementation: "shell code is compliant iff it runs without errors under this implementation" (*). That implementation could be bash in POSIX mode, meaning that e.g. '[[ ]]' would be allowed. Alternatively, it could be a minimal shell like "pbosh". So far the discussion in this thread did not come up with a prime candidate for being that reference shell. dash has been propoposed and then harshly dismissed. ash, likewise. I suppose that Jörg would propose "pbosh", and it seems to come closer to the ideal of a "strict" POSIX shell as anything else mentioned up to now. But I'd call it a bit exotic. It doesn't seem to have a lot of real-world users under openSUSE. I still fail to see the benefit of choosing anything else than bash in POSIX mode for /bin/sh, unless we want to pursue the long-term goal to be able to run (a minimal) openSUSE completely without bash. Regards, Martin (#) that doesn't mean we actually have to test every shell code snippet, but we _could_ in case of disagreement. (*) ... and yields correct results for a reasonable set of possible inputs. -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 2019-02-18 18:46, Martin Wilck wrote:
We need to define without ambiguity under what conditions a _shell script_ complies with the feature set of "/bin/sh". That means that we need to have a _test procedure_ (#). We could say that "code is compliant iff it runs without errors under every POSIX-compliant shell (*)". But that's hard to test
With a single prjconf line, we can make Factory (in a cloned project) build everything with an alternate sh. Haven't yet come to a selection whether or not I should do the initial build on a private instance, or start outright on build.opensuse.org. For essential runtime testing, there is OpenQA. Perhaps it's possible to add a job that derives from an existing run and has an extra action that switches sh via update-alternatives after installation, and then runs the rest of the testsuite. I don't quite know if OpenQA is capable of that yet. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19/02/2019 04:52, Jan Engelhardt wrote:
On Monday 2019-02-18 18:46, Martin Wilck wrote:
We need to define without ambiguity under what conditions a _shell script_ complies with the feature set of "/bin/sh". That means that we need to have a _test procedure_ (#). We could say that "code is compliant iff it runs without errors under every POSIX-compliant shell (*)". But that's hard to test
With a single prjconf line, we can make Factory (in a cloned project) build everything with an alternate sh. Haven't yet come to a selection whether or not I should do the initial build on a private instance, or start outright on build.opensuse.org.
For essential runtime testing, there is OpenQA. Perhaps it's possible to add a job that derives from an existing run and has an extra action that switches sh via update-alternatives after installation, and then runs the rest of the testsuite. I don't quite know if OpenQA is capable of that yet.
I think testing in openQA is probably too late unless we get to a point where we can reject submit requests based on its output. Unless your interested in filing bugs rather then blocking non compliant packages. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19/02/2019 04:16, Martin Wilck wrote:
On Mon, 2019-02-18 at 13:36 +0100, Joerg Schilling wrote:
Mischa Salle <mischa.salle@gmail.com> wrote:
I might have missed part of this (very long) discussion, but in my opinion - using /bin/sh means you SHOULD only use constructions which are POSIX compliant. - when you want to use bashisms, use /bin/bash. The same applies to other shells.
This is finally the right conclusion even thouh POSIX does not make any statements on what /bin/sh is.
I don't think that this is the right conclusion. Here's why.
I've been told in this discussion that the POSIX shell standard for "allows extensions". IOW, it only defines a minimal set of features, and implementations are free to add more features on top and still "comply" with the standard. Therefore the fact that bash, invoked as "/bin/sh", supports the non-POSIX '[[ ]]' syntax doesn't make it non- compliant.
We need to define without ambiguity under what conditions a _shell script_ complies with the feature set of "/bin/sh". That means that we need to have a _test procedure_ (#). We could say that "code is compliant iff it runs without errors under every POSIX-compliant shell (*)". But that's hard to test, as we'd have to test with every POSIX- compliant shell out there, and we may not even know them all. We also don't want to test by human beings parsing the code (even if that's basically what Jan is doing today). Rather, we need to settle on an actual shell implementation, and define "compliant" by reference to this implementation: "shell code is compliant iff it runs without errors under this implementation" (*).
That implementation could be bash in POSIX mode, meaning that e.g. '[[ ]]' would be allowed. Alternatively, it could be a minimal shell like "pbosh".
So far the discussion in this thread did not come up with a prime candidate for being that reference shell. dash has been propoposed and then harshly dismissed. ash, likewise. I suppose that Jörg would propose "pbosh", and it seems to come closer to the ideal of a "strict" POSIX shell as anything else mentioned up to now. But I'd call it a bit exotic. It doesn't seem to have a lot of real-world users under openSUSE.
I still fail to see the benefit of choosing anything else than bash in POSIX mode for /bin/sh, unless we want to pursue the long-term goal to be able to run (a minimal) openSUSE completely without bash.
Regards, Martin
(#) that doesn't mean we actually have to test every shell code snippet, but we _could_ in case of disagreement. (*) ... and yields correct results for a reasonable set of possible inputs.
I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features. Down the road if we decide as a project that we want to officially support multiple shells as #!/bin/sh I'd like to see debians checkers intergrated into rpmlint by the people who want to see this feature. I also don't think we need to have and agree on one explicit reference shell when I'm working on xdg-utils I generally go with use a subset of functionality thats in both bash and dash, maybe in openSUSE's case someone can make a compelling argument that pbosh should be added to the list then thats a case for people who are actually working on making it possible and wanting to run pbosh as #!/bin/sh to make. Not something we should add because some people believe it is the closest thing to the posix standard. We have to remember that the vast majority of the scripts we are talking about here are openSUSE / SUSE specific install scripts that wont make sense to run outside an openSUSE system and don't really need to be portable to the unix's of a bygone era. If the scripts don't fit that category then really they should be fixed wherever there upstream is. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, 2019-02-19 at 13:24 +1030, Simon Lees wrote:
I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features.
So - encourage people to remove bashisms on the one hand, and on the other hand allow more bashisms to creep in? As a policy, that doesn't make much sense to me. We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged. If we go for b), we must avoid bashisms in new submissions. Even the syntax "%post -p /bin/bash" should be considered transitional in that case. IMO it has to be either a) or b) eventually. I see no benefit in looking for a middle ground (other than for transition).
Down the road if we decide as a project that we want to officially support multiple shells as #!/bin/sh I'd like to see debians checkers intergrated into rpmlint by the people who want to see this feature.
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do. Regards Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Feb 19, 2019 at 11:09 AM Martin Wilck <mwilck@suse.com> wrote:
On Tue, 2019-02-19 at 13:24 +1030, Simon Lees wrote:
I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features.
So - encourage people to remove bashisms on the one hand, and on the other hand allow more bashisms to creep in? As a policy, that doesn't make much sense to me.
We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged.
I see no point in going the Debian route. Forcing people to write scriptlets purely in POSIX is just adding pain for no benefit. The _only_ reason Debian enforces it is because they have a case where their init is a shell script. And even then, it's easy enough to argue otherwise since Bash continues to evolve and improve. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tuesday 2019-02-19 17:12, Neal Gompa wrote:
it's easy enough to argue otherwise since Bash continues to "evolve" and "improve".
[emphasis mine] The only question is... in what direction. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Feb 19, 2019 at 5:13 PM Neal Gompa <ngompa13@gmail.com> wrote:
We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged.
I see no point in going the Debian route. Forcing people to write scriptlets purely in POSIX is just adding pain for no benefit. The _only_ reason Debian enforces it is because they have a case where their init is a shell script. And even then, it's easy enough to argue otherwise since Bash continues to evolve and improve.
No one is suggesting you have to write POSIX-compliant scripts, except in the case when you use #!/bin/sh. If you want to use any-other-shell-specifics, then specify that shell instead. In any case I would certainly go for option b) which allows the freedom to symlink /bin/sh to a faster performing or more POSIX compliant shell. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Martin Wilck schrieb:
On Tue, 2019-02-19 at 13:24 +1030, Simon Lees wrote:
I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features.
So - encourage people to remove bashisms on the one hand, and on the other hand allow more bashisms to creep in? As a policy, that doesn't make much sense to me.
We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged. If we go for b), we must avoid bashisms in new submissions. Even the syntax "%post -p /bin/bash" should be considered transitional in that case.
IMO it has to be either a) or b) eventually. I see no benefit in looking for a middle ground (other than for transition).
Down the road if we decide as a project that we want to officially support multiple shells as #!/bin/sh I'd like to see debians checkers intergrated into rpmlint by the people who want to see this feature.
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 3/12/19 3:41 PM, Ludwig Nussel wrote:
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Mar 12, 2019 at 10:41 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Martin Wilck schrieb:
On Tue, 2019-02-19 at 13:24 +1030, Simon Lees wrote:
I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features.
So - encourage people to remove bashisms on the one hand, and on the other hand allow more bashisms to creep in? As a policy, that doesn't make much sense to me.
We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged. If we go for b), we must avoid bashisms in new submissions. Even the syntax "%post -p /bin/bash" should be considered transitional in that case.
IMO it has to be either a) or b) eventually. I see no benefit in looking for a middle ground (other than for transition).
Down the road if we decide as a project that we want to officially support multiple shells as #!/bin/sh I'd like to see debians checkers intergrated into rpmlint by the people who want to see this feature.
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
That creates a dependency loop, though. Managing the core shell through alternatives is nuts because it'll break things for stateless things, require a scriptlet engine to run the program, and the scriptlet engine isn't configured yet because alternatives isn't set up... -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Mar 12, Neal Gompa wrote:
On Tue, Mar 12, 2019 at 10:41 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
That creates a dependency loop, though. Managing the core shell through alternatives is nuts because it'll break things for stateless things, require a scriptlet engine to run the program, and the scriptlet engine isn't configured yet because alternatives isn't set up...
The loop is only there because of the post install section of update-alternatives. Somebody needs to implement that in lua ;) Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, 2019-03-12 at 15:51 +0100, Stephan Kulow wrote:
On 3/12/19 3:41 PM, Ludwig Nussel wrote:
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it
No it is not. A public discussion which was by no means settled is being ended the hard way, "creating facts". I can't imagine that that's what Richard meant. This "solution" is thoroughly lacking the "human touch" that Richard mentioned in his talk. It's also highly doubtful whether everyone involved in the discussion "feels they were heard". I, personally, do not. My key point in the discussion was that we need to settle on a unique, well-defined, existing shell with which scripts can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out. What I least understand is that this massive change is being rushed to factory in less than 2 weeks, while other things keep lurking in home projects or devel projects for ages. For example, in the course of this discussion, I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123). That was my approach to the subject, take some small steps, evaluate benefits, risks, and challenges, and then flip the switch (or not). But the big axe seems to be preferred. Well then. I've learned a not-so- nice lesson about openSUSE today. I probably sound like a sore looser. Perhaps I am, although I'm not strongly on one side of the "pro-and-contra bash" discussion. I understand the "openSUSE is what you make it" mantra. But I dislike the way it's bluntly being applied here. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 07:28, Martin Wilck wrote:
On Tue, 2019-03-12 at 15:51 +0100, Stephan Kulow wrote:
On 3/12/19 3:41 PM, Ludwig Nussel wrote:
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it
No it is not. A public discussion which was by no means settled is being ended the hard way, "creating facts". I can't imagine that that's what Richard meant. This "solution" is thoroughly lacking the "human touch" that Richard mentioned in his talk. It's also highly doubtful whether everyone involved in the discussion "feels they were heard". I, personally, do not. My key point in the discussion was that we need to settle on a unique, well-defined, existing shell with which scripts can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
What I least understand is that this massive change is being rushed to factory in less than 2 weeks, while other things keep lurking in home projects or devel projects for ages. For example, in the course of this discussion, I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).
That was my approach to the subject, take some small steps, evaluate benefits, risks, and challenges, and then flip the switch (or not). But the big axe seems to be preferred. Well then. I've learned a not-so- nice lesson about openSUSE today.
I probably sound like a sore looser. Perhaps I am, although I'm not strongly on one side of the "pro-and-contra bash" discussion. I understand the "openSUSE is what you make it" mantra. But I dislike the way it's bluntly being applied here.
I tend to agree, with Martin here, I don't think such a change should be made until we can guarantee things wont break, and we can't do that until there is a way for packagers like me who have never cared what is / isn't a bashism to know they are breaking something as part of the buildsystem. I would request that those who are making such SR's stop until there is checks for bashism's as part of the buildsystem, otherwise i'll raise it to the board, as quite clearly at the moment the objections / concerns raised in the discussions on this list are not being taken into account in the current submissions. While I am generally ok with people making this change if they would like to I think it needs to be done right, and the current submission is not doing it right. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 07:28, Martin Wilck wrote:
On Tue, 2019-03-12 at 15:51 +0100, Stephan Kulow wrote:
On 3/12/19 3:41 PM, Ludwig Nussel wrote:
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it
No it is not. A public discussion which was by no means settled is being ended the hard way, "creating facts". I can't imagine that that's what Richard meant. This "solution" is thoroughly lacking the "human touch" that Richard mentioned in his talk. It's also highly doubtful whether everyone involved in the discussion "feels they were heard". I, personally, do not. My key point in the discussion was that we need to settle on a unique, well-defined, existing shell with which scripts can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
What I least understand is that this massive change is being rushed to factory in less than 2 weeks, while other things keep lurking in home projects or devel projects for ages. For example, in the course of this discussion, I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).
That was my approach to the subject, take some small steps, evaluate benefits, risks, and challenges, and then flip the switch (or not). But the big axe seems to be preferred. Well then. I've learned a not-so- nice lesson about openSUSE today.
I probably sound like a sore looser. Perhaps I am, although I'm not strongly on one side of the "pro-and-contra bash" discussion. I understand the "openSUSE is what you make it" mantra. But I dislike the way it's bluntly being applied here.
I tend to agree, with Martin here, I don't think such a change should be made until we can guarantee things wont break, and we can't do that until there is a way for packagers like me who have never cared what is / isn't a bashism to know they are breaking something as part of the buildsystem.
I would request that those who are making such SR's stop until there is checks for bashism's as part of the buildsystem, otherwise i'll raise it to the board, as quite clearly at the moment the objections / concerns raised in the discussions on this list are not being taken into account in the current submissions.
While I am generally ok with people making this change if they would like to I think it needs to be done right, and the current submission is not doing it right. Though not affected myself, I tend to agree with Martin as well. What about an
Op dinsdag 12 maart 2019 22:58:07 CET schreef Simon Lees: option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells? -- Gertjan Lettink a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 08:33, Knurpht-openSUSE wrote:
On 13/03/2019 07:28, Martin Wilck wrote:
On Tue, 2019-03-12 at 15:51 +0100, Stephan Kulow wrote:
On 3/12/19 3:41 PM, Ludwig Nussel wrote:
The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it
No it is not. A public discussion which was by no means settled is being ended the hard way, "creating facts". I can't imagine that that's what Richard meant. This "solution" is thoroughly lacking the "human touch" that Richard mentioned in his talk. It's also highly doubtful whether everyone involved in the discussion "feels they were heard". I, personally, do not. My key point in the discussion was that we need to settle on a unique, well-defined, existing shell with which scripts can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
What I least understand is that this massive change is being rushed to factory in less than 2 weeks, while other things keep lurking in home projects or devel projects for ages. For example, in the course of this discussion, I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).
That was my approach to the subject, take some small steps, evaluate benefits, risks, and challenges, and then flip the switch (or not). But the big axe seems to be preferred. Well then. I've learned a not-so- nice lesson about openSUSE today.
I probably sound like a sore looser. Perhaps I am, although I'm not strongly on one side of the "pro-and-contra bash" discussion. I understand the "openSUSE is what you make it" mantra. But I dislike the way it's bluntly being applied here.
I tend to agree, with Martin here, I don't think such a change should be made until we can guarantee things wont break, and we can't do that until there is a way for packagers like me who have never cared what is / isn't a bashism to know they are breaking something as part of the buildsystem.
I would request that those who are making such SR's stop until there is checks for bashism's as part of the buildsystem, otherwise i'll raise it to the board, as quite clearly at the moment the objections / concerns raised in the discussions on this list are not being taken into account in the current submissions.
While I am generally ok with people making this change if they would like to I think it needs to be done right, and the current submission is not doing it right. Though not affected myself, I tend to agree with Martin as well. What about an
Op dinsdag 12 maart 2019 22:58:07 CET schreef Simon Lees: option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?
My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new (bashism vs normal shell) in order to not break things. Personally I think asking other people to learn new skills to care about things for you is not the best approach and doesn't fit with the openSUSE "Those who do decide" as lots of maintainers can decide not to learn and then we end up with something that is broken, whereas if a bot tells me i've screwed up in the same way a compiler does then its minimal effort for me to go and fix it. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tuesday 2019-03-12 17:23, Neal Gompa wrote:
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
That creates a dependency loop, though.
bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.
Managing the core shell through alternatives is nuts because it'll break things for stateless things, require a scriptlet engine to run the program
A trivial look shows that update-alternatives is an ELF binary. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tuesday 2019-03-12 23:46, Simon Lees wrote:
Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?
My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new
My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box. Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month. [insert scared pikachu meme image here] -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 09:53, Jan Engelhardt wrote:
On Tuesday 2019-03-12 23:46, Simon Lees wrote:
Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?
My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new
My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box.
Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month.
So your adding an update-alternatives option that allows people to break there systems? I don't think that's any better. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
* Simon Lees <sflees@suse.de> [03-12-19 19:31]:
On 13/03/2019 09:53, Jan Engelhardt wrote:
On Tuesday 2019-03-12 23:46, Simon Lees wrote:
Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?
My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new
My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box.
Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month.
So your adding an update-alternatives option that allows people to break there systems? I don't think that's any better.
it is not a requirement to utilize update-alternatives in order to break one's system. if one does so in that manner, he has chosen do do so and you will not stop him by taking away that option. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Op woensdag 13 maart 2019 00:23:10 CET schreef Jan Engelhardt:
On Tuesday 2019-03-12 23:46, Simon Lees wrote:
Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?
My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new
My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box.
Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month.
[insert scared pikachu meme image here] Hey, Jan I'm just trying to bring up ideas.
-- Gertjan Lettink a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 10:19, Patrick Shanahan wrote:
* Simon Lees <sflees@suse.de> [03-12-19 19:31]:
On 13/03/2019 09:53, Jan Engelhardt wrote:
On Tuesday 2019-03-12 23:46, Simon Lees wrote:
Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?
My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new
My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box.
Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month.
So your adding an update-alternatives option that allows people to break there systems? I don't think that's any better.
it is not a requirement to utilize update-alternatives in order to break one's system. if one does so in that manner, he has chosen do do so and you will not stop him by taking away that option.
If openSUSE is going to offer this as a supported option ie using update-alternatives rather then manually changing the symlink then we need to make sure it works, its all good for me and you to know it might make a mess of our systems but a general user who is not reading this list might see that the option has been added and make the reasonable presumption that it is supported and works which could lead to them breaking there system which is unacceptable. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 13.03.19 um 01:36 schrieb Simon Lees:
If openSUSE is going to offer this as a supported option
What does it bring if openSUSE supports an option? People file bug reports against packages - and maintainers of those close them WONTFIX as "send a patch, I don't support this myself". So please don't make more out of it as it is - it's not as your openSUSE salary depends on happy openSUSE users. First of all to break your system with this, other shells need to provide alternatives as well - and it's up to these shells to make sure their description says DO NOT EXPECT THIS TO WORK IN ALL USE CASES. Greetings, Stephan -- Lighten up, just enjoy life, smile more, laugh more, and don't get so worked up about things. Kenneth Branagh -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 12.03.19 um 21:58 schrieb Martin Wilck:
I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).
Please bring this up a little earlier - we have `factory-maintainers` group to help with devel project maintainers not being active anymore. Happens in all communities that contributors drop out - and we have measures not to make this a big problem. Greetings, Stephan -- Lighten up, just enjoy life, smile more, laugh more, and don't get so worked up about things. Kenneth Branagh -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 13.03.19 um 00:16 schrieb Jan Engelhardt:
A trivial look shows that update-alternatives is an ELF binary.
but %{pre,post,preun,postun,...} are executed with the help of a shell, That's the whole point of this pointless thread IIRC. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mär 13 2019, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
but %{pre,post,preun,postun,...} are executed with the help of a shell
Not necessarily. They can also be implemented in embedded lua. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Martin Wilck <mwilck@suse.com> wrote:
can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
I doubt that. Currently, you have some obscure "wisdom" that there is something called "bash" where nobody is able to tell what features that covers since there is more than one single release of bash. If you however write POSIX compliant scripts, you have the grant that in case there is a problem, you can file a bug against the installed shell. For me, this looks like a big win. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Engelhardt <jengelh@inai.de> wrote:
On Tuesday 2019-03-12 17:23, Neal Gompa wrote:
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
That creates a dependency loop, though.
bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.
I am not sure what you mean by that, but bash can be compiled as "POSIX compliant". This is e.g. done on Mac OS (where bash is installed as /bin/sh and where no other POSIX shell is provided by the OS) and on Solaris, where bash is just installed as /usr/bin/bash (with a symlink /usr -> /usr/bin) and where bash is just one of many shells. If you do that, you break all noncompliant scripts that typically make non-POSIX assumptions on the behavior of the built in "echo" command. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Mar 13, 2019 at 6:21 AM Joerg Schilling <Joerg.Schilling@fokus.fraunhofer.de> wrote:
Martin Wilck <mwilck@suse.com> wrote:
can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
I doubt that.
Currently, you have some obscure "wisdom" that there is something called "bash" where nobody is able to tell what features that covers since there is more than one single release of bash.
If you however write POSIX compliant scripts, you have the grant that in case there is a problem, you can file a bug against the installed shell.
For me, this looks like a big win.
This is not a win. This is a straight up fail. If you do this, you are allowing it to be possible to have a broken openSUSE bootstrap. You're allowing for people to choose an unsupported configuration in the very core of the distribution. That is completely nuts. Now admittedly, I don't like how much openSUSE uses alternatives for everything already, but at least we're not using it in ways where you'd potentially break your computer with a choice. This is literally making it so it's possible to have a failed openSUSE install. And even if the scriptlet was in Lua, there's other transactional ordering issues that would cause problems. It's much saner to just have bash as the supported /bin/sh and declare it as such. If you want to use a different shell for a scriptlet, specify it. This is insane. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Neal Gompa <ngompa13@gmail.com> wrote:
This is not a win. This is a straight up fail. If you do this, you are allowing it to be possible to have a broken openSUSE bootstrap. You're allowing for people to choose an unsupported configuration in the very core of the distribution. That is completely nuts.
I doubt that the bootstrap may fail _because_ you wrote POSIX compliant scripts except when there is a non-POSIX-compliant default shell and scripts use that shell. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 20:55, Joerg Schilling wrote:
Jan Engelhardt <jengelh@inai.de> wrote:
On Tuesday 2019-03-12 17:23, Neal Gompa wrote:
I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.
That creates a dependency loop, though.
bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.
I am not sure what you mean by that, but bash can be compiled as "POSIX compliant". This is e.g. done on Mac OS (where bash is installed as /bin/sh and where no other POSIX shell is provided by the OS) and on Solaris, where bash is just installed as /usr/bin/bash (with a symlink /usr -> /usr/bin) and where bash is just one of many shells.
If you do that, you break all noncompliant scripts that typically make non-POSIX assumptions on the behavior of the built in "echo" command.
Given that posix has extensions and all the shells have some wriggle room to do things slightly differently, having something like the subset of features that work on shell X, Y and Z is far more useful and actually testable where if you just say "posix" people could choose to interpret that as bash --posix which won't necessarily work on dash so the point is we need to create an openSUSE definition of posix that is clear and everyone understands such a definition could be "the subset of commands / syntax that works on both bash and dash", if there are other shells that people care about that break under that definition its simple to add them. Then everything is clear and easy bots can be written etc... -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 17:10, Stephan Kulow wrote:
Am 13.03.19 um 01:36 schrieb Simon Lees:
If openSUSE is going to offer this as a supported option
What does it bring if openSUSE supports an option? People file bug reports against packages - and maintainers of those close them WONTFIX as "send a patch, I don't support this myself".
Personally I don't think people should have to file potentially high severity bugs when it is not hard to prevent developers introducing said bugs in the first place and when fixing a bug will only ever take 1 line ie executing the script in bash rather then /bin/sh it is really not that much to expect them to fix it (currently the review team expects far more in other areas)
So please don't make more out of it as it is - it's not as your openSUSE salary depends on happy openSUSE users. First of all to break your system with this,
Well its a good thing my salary depends on happy SLE users instead, but the reality is I care about the quality of openSUSE, and I will call out developers who compromise the quality of openSUSE by taking lazy shortcuts rather then doing things properly.
other shells need to provide alternatives as well - and it's up to these shells to make sure their description says DO NOT EXPECT THIS TO WORK IN ALL USE CASES.
Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 3/13/19 1:44 PM, Simon Lees wrote:
Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.
It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux. Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 13 Mar 2019 11:20:28 +0100 Joerg Schilling <Joerg.Schilling@fokus.fraunhofer.de> wrote:
Martin Wilck <mwilck@suse.com> wrote:
can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
I doubt that.
Currently, you have some obscure "wisdom" that there is something called "bash" where nobody is able to tell what features that covers since there is more than one single release of bash.
If you however write POSIX compliant scripts, you have the grant that in case there is a problem, you can file a bug against the installed shell.
Just as POSIX is a description of disparate implementations with various issues and the standard is updated over time as well. Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently. So from compatibility and reliability standpoint bash is clear win if you are in control of the shell used. And that is our case so long as we do not implement alternatives for /bin/sh. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 3/13/19 1:56 PM, Michal Suchánek wrote:
Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.
And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow <coolo@suse.de> wrote:
On 3/13/19 1:56 PM, Michal Suchánek wrote:
Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.
And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build. Anyway, if we provide alternatives for /bin/sh you can assume any shell that claims to be vaguely bourne-compatible can end up as /bin/sh. All these shells have various bugs. Writing shell code that is bug-compatible with all shells the distribution ships is non-trivial. Some bugs are due to parser issues that are complex to fix and will not be resolved overnight. Ergo if we do provide alternatives for /bin/sh we cannot expect anything from /bin/sh anymore. Then you need to specify the interpreter for every shell script. Then there is no point in even having a /bin/sh. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 3/13/19 2:19 PM, Michal Suchánek wrote:
On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow <coolo@suse.de> wrote:
On 3/13/19 1:56 PM, Michal Suchánek wrote:
Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.
And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.Again, if you find yourself in need to write bash version specific %build scripts, write them in a real language - or at least put them in an extra script.
Anyway, if we provide alternatives for /bin/sh you can assume any shell
Yeah, that's right. *If* - *if* a shell in openSUSE wants to provide /bin/sh, it needs to eat all our rpm snippets that survived checkbashisms. Just because bash provides it, doesn't mean there are alternatives worth fighting over theoretic problems. And %build sections should potentially even be executed by /bin/bash --posix right away. But again, if your %build section is bash version specific, it's a lost case IMO. Greetings, Stephan -- A good name lost is seldom regained. When character is gone, all is gone, and one of the richest jewels of life is lost forever. -- J. Hawes -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/03/2019 23:23, Stephan Kulow wrote:
On 3/13/19 1:44 PM, Simon Lees wrote:
Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.
It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.
My point is more that I don't find it an acceptable acceptance level for openSUSE particularly when the amount of extra effort to make it much more reliable isn't large, but I guess we can agree to disagree.
Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.
Sure you can experiment with this it would take about 2 minutes and not much knowledge to create a repo in someones home dir where dash provided #!/bin/sh for example and bash did not once done they could even document it in a blog post for others to mess with. To me thats where this level of proposal belongs. On the other hand if one was to write an rpmlint style check to catch bashism's in #!/bin/sh code and then fixed the resulting errors i'd be happy to have the update-alternatives in openSUSE because the chances of it breaking systems would be significantly lower. Given such scripts already exist in debian land I don't think this is too much work and would provide a much more proper solution. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2019-03-13 14:19:23 +0100, Michal Suchánek wrote:
On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow <coolo@suse.de> wrote:
On 3/13/19 1:56 PM, Michal Suchánek wrote:
Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.
And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.
You can specify it via the %__spec_build_cmd macro. For instance, you could add a %define __spec_build_cmd /bin/bash -e line to your spec file. Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 2019-03-13 at 11:20 +0100, Joerg Schilling wrote:
Martin Wilck <mwilck@suse.com> wrote:
can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.
I doubt that.
Currently, you have some obscure "wisdom" that there is something called "bash" where nobody is able to tell what features that covers since there is more than one single release of bash.
I don't disagree. This is one argument for using a different shell, which is in "feature freeze" mode. OTOH, the changes between bash 4.x and 5.0 are rather insignificant in terms of shell script syntax, AFAICT.
If you however write POSIX compliant scripts, you have the grant that in case there is a problem, you can file a bug against the installed shell.
For me, this looks like a big win.
I for one wouldn't want to rely on my brain's built-in Posix shell code parser. I'd much prefer to have a computer do it for me. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 2019-03-13 at 22:58 +1030, Simon Lees wrote:
the point is we need to create an openSUSE definition of posix that is clear and everyone understands such a definition could be "the subset of commands / syntax that works on both bash and dash",
I'd rather settle down on _one_ shell to test with. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 2019-03-13 at 13:53 +0100, Stephan Kulow wrote:
Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.
I'd find this acceptable if reviewers *vow* not to let another shell providing a "/bin/sh" alternative enter the official factory repo for some time to come (a year, say). That would open up the possibility for adventurous people to experiment with other shells in their projects, without overriding factory's bash package. Then, if someone can prove that their project is doing well and perhaps even build times are reduced because of the faster speed of their alternative shell, we could re-evaluate the whole thing. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Martin Wilck <mwilck@suse.com> wrote:
I don't disagree. This is one argument for using a different shell, which is in "feature freeze" mode. OTOH, the changes between bash 4.x and 5.0 are rather insignificant in terms of shell script syntax, AFAICT.
I remember that we recently discoverd a significant difference between bash4 and bash4 while discussing shell behavior in the POSIX standard teleconference. Sorry, I no longer remember what exactly this was related to.
I for one wouldn't want to rely on my brain's built-in Posix shell code parser. I'd much prefer to have a computer do it for me.
I do it this way: I primarily believe that the parser in bosh is correct, since it eveolved since more than 42 years. Whenever I see a problem, I check the behavior of ksh88, ksh93 and bosh and then decide using my brain what is correct. The trick behind that method is that all three shells are direct descendents from the original Bourne Shell (which is a good starter) and their maintainers tried to do their best to fix problems in the behavior of the original Bourne Shell. Writing a correct shell is definitely not trivial and other shells that have been written from scratch usually have more or less deviations. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Joerg Schilling <Joerg.Schilling@fokus.fraunhofer.de> wrote:
Martin Wilck <mwilck@suse.com> wrote:
I don't disagree. This is one argument for using a different shell, which is in "feature freeze" mode. OTOH, the changes between bash 4.x and 5.0 are rather insignificant in terms of shell script syntax, AFAICT.
I remember that we recently discoverd a significant difference between bash4 and bash4 while discussing shell behavior in the POSIX standard teleconference. Sorry, I no longer remember what exactly this was related to.
Sorry for the typo bash4 and bash5 was correct. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 13 Mar 2019 14:30:36 +0100 Stephan Kulow <coolo@suse.de> wrote:
On 3/13/19 2:19 PM, Michal Suchánek wrote:
On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow <coolo@suse.de> wrote:
On 3/13/19 1:56 PM, Michal Suchánek wrote:
Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.
And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.Again, if you find yourself in need to write bash version specific %build scripts, write them in a real language - or at least put them in an extra script.
I am fine with it being any bash version form SLE11 to TW. As in if it builds everywhere it's ok. I am not fine if we switch it to 'any shell for which somebody installs an alternative'.
Anyway, if we provide alternatives for /bin/sh you can assume any shell
Yeah, that's right. *If* - *if* a shell in openSUSE wants to provide /bin/sh, it needs to eat all our rpm snippets that survived checkbashisms. Just because bash provides it, doesn't mean there are alternatives worth fighting over theoretic problems.
That's the problem. If people provide a broken /bin/sh alternative or one of the /bin/sh alternatives eventually regresses people will come complaining to me that my package does not build. If people provide two different /bin/sh alternatives with very different behavior in some corner cases I will need to rewrite my packaging scripts when I hit the corner case. I have been there writing scripts for Debian which provides several /bin/sh alternatives each with its unique set of bugs. I want no more of that.
And %build sections should potentially even be executed by /bin/bash --posix right away. But again, if your %build section is bash version specific, it's a lost case IMO.
You need not even use bash-specific syntax at all. You might just use some variable scoping that happens to work in bash and happens to not work in some other shell. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 13 Mar 2019 14:54:03 +0100 Marcus Hüwe <suse-tux@gmx.de> wrote:
On 2019-03-13 14:19:23 +0100, Michal Suchánek wrote:
On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow <coolo@suse.de> wrote:
On 3/13/19 1:56 PM, Michal Suchánek wrote:
Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.
And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.
You can specify it via the %__spec_build_cmd macro. For instance, you could add a
%define __spec_build_cmd /bin/bash -e
Yes, that's helpful. If we get these alternatives I will add it to the compatibility cruft block together with the %license handling. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wednesday 2019-03-13 09:50, Andreas Schwab wrote:
On Mär 13 2019, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
but %{pre,post,preun,postun,...} are executed with the help of a shell
Not necessarily. They can also be implemented in embedded lua.
And we could also just (maybe) write %pre -p /usr/sbin/update-alternatives --install ... Exciting how rpm works. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wednesday 2019-03-13 11:25, Joerg Schilling wrote:
bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.
I am not sure what you mean by that
The build log for bash.rpm showed that "update-alternatives" was already installed as a build-time dependency to create openSUSE's bash. So the particular situation that a build cycle already existed was not made worse. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 14/03/2019 00:52, Martin Wilck wrote:
On Wed, 2019-03-13 at 22:58 +1030, Simon Lees wrote:
the point is we need to create an openSUSE definition of posix that is clear and everyone understands such a definition could be "the subset of commands / syntax that works on both bash and dash",
I'd rather settle down on _one_ shell to test with.
Martin
That indeed would be nice, but I don't think its really possible without writing another reference shell (or forking an existing one to remove features until we get close enough). However there exists checkers that catch most if not all of the common errors and having them setup in the build process to do the majority of testing automatically (so in general packagers need to do nothing) seems like a better approach, then in case someone really wants to argue semantics we can point them to does the script run in bash and dash, if yes its acceptable if not its not for example. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 14, 2019 at 12:01:16AM +0100, Jan Engelhardt wrote:
On Wednesday 2019-03-13 09:50, Andreas Schwab wrote:
On Mär 13 2019, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
but %{pre,post,preun,postun,...} are executed with the help of a shell
Not necessarily. They can also be implemented in embedded lua.
And we could also just (maybe) write
%pre -p /usr/sbin/update-alternatives --install ...
Exciting how rpm works.
It should be mentioned that with e.g. %post -p <lua> the os.execute() and io.popen() lua API do use /bin/sh just like (g)libc system(3), that is ther is no os.execve() :P Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
Stephan Kulow schrieb:
On 3/13/19 1:44 PM, Simon Lees wrote:
Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.
It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.
Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.
If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage. On package level we have better options to control what goes where and what gets installed by default, eg by means of Requires in patterns or NON_FTP_PACKAGES. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 14, Jan Engelhardt wrote:
On Wednesday 2019-03-13 11:25, Joerg Schilling wrote:
bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.
I am not sure what you mean by that
The build log for bash.rpm showed that "update-alternatives" was already installed as a build-time dependency to create openSUSE's bash. So the particular situation that a build cycle already existed was not made worse.
The problem is the installation dependency cycle: - bash requires update-alternatve - update-alternative requires bash If you install bash first, the update-alternative call in the %post section will fail and there is no /bin/sh. If you install update-alternative first, the %post will fail. If this is a fresh installation, it luckily does not matter, but this may generate a failure and you will have a inconsistent system and RPM database afterwards. What I'm still missing is: what is the advantage to be able to replace /bin/sh with something different than bash? bash will be installed anyways, there is no way to install an openSUSE without bash, there are too many core packages requiring /bin/bash. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 14, 2019 at 08:13:44AM +0100, Ludwig Nussel wrote:
Stephan Kulow schrieb:
On 3/13/19 1:44 PM, Simon Lees wrote:
Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.
It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.
Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.
If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage. On package level we have better options to control what goes where and what gets installed by default, eg by means of Requires in patterns or NON_FTP_PACKAGES.
This I've suggested in the submit request comments, that is that bash.spec provides a sub package which only includes the link /bin/sh -> /usr/bin/sh even if I do not like those links between Essential User Binaries and the Non-essential User Binaries. Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
On Wed, 2019-03-13 at 18:43 +0100, Michal Suchánek wrote:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a
%define __spec_build_cmd /bin/bash -e
Yes, that's helpful.
If we get these alternatives I will add it to the compatibility cruft block together with the %license handling.
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section. The vast majoriti of packages uses "configure, make, makeinstall", "%cmake, %cmake_build,, %cmake_install"… The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something. Just saying - Having a way out is good, opposing it for the sake of opposing it is unfriendly in a community. Cheers, Dominique PS: the official way of building packages for openSUSE and SLE will stay with osc build and OBS - And of course, should anybody ever have the idea of switching the defaulit shell in this setup, you can be sure that this won't happen without a staging pass.
On Thu, 2019-03-14 at 08:41 +0100, Dominique Leuenberger / DimStar wrote:
Just saying - Having a way out is good, opposing it for the sake of opposing it is unfriendly in a community.
The same can be said about pushing through a request that lots of people are opposing, just "because we can". Regards, Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mär 14 2019, Jan Engelhardt <jengelh@inai.de> wrote:
And we could also just (maybe) write
%pre -p /usr/sbin/update-alternatives --install ...
Exciting how rpm works.
I can't pass arguments to the interpreter here. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Simon Lees <sflees@suse.de> wrote:
That indeed would be nice, but I don't think its really possible without writing another reference shell (or forking an existing one to remove features until we get close enough). However there exists checkers that catch most if not all of the common errors and having them setup in the build process to do the majority of testing automatically (so in general packagers need to do nothing) seems like a better approach, then in case someone really wants to argue semantics we can point them to does the script run in bash and dash, if yes its acceptable if not its not for example.
bosh comes with a fine grained #ifdef structure that allows to compile it to behave like the Solaris SVr4 based Bourne Shell up to the POSIX variant with all it's enhencements, so this is nothing you need to invent or develop just for this purpose. Check the Makefile for explanations. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 14, 2019 at 08:41:24AM +0100, Dominique Leuenberger / DimStar wrote:
On Wed, 2019-03-13 at 18:43 +0100, Michal Suchánek wrote:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a
%define __spec_build_cmd /bin/bash -e
Yes, that's helpful.
If we get these alternatives I will add it to the compatibility cruft block together with the %license handling.
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
The vast majoriti of packages uses "configure, make, makeinstall", "%cmake, %cmake_build,, %cmake_install"…
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Just saying - Having a way out is good, opposing it for the sake of opposing it is unfriendly in a community.
I vote for configure the system Bourne SHell on the command line. As an example I provide the new package posh (Policy-compliant Ordinary SHell) in the project shells. With this the System Administrator has to do update-alternatives --config sh which gives There are 2 choices for the alternative sh (providing /usr/bin/sh). Selection Path Priority Status ------------------------------------------------------------ * 0 /usr/bin/bash 10100 auto mode 1 /usr/bin/bash 10100 manual mode 2 /usr/bin/posh 100 manual mode Press <enter> to keep the current choice[*], or type selection number: 2 update-alternatives: using /usr/bin/posh to provide /usr/bin/sh (sh) in manual mode and this works. Only the symbolic link /usr/bin/sh should not be part of the file list regardless what rpmlint says here as with uninstalling the package the link /usr/bin/sh should stay!! The difference between bash.spec and posh.spec is that I use in the first spec file the option --force for u-a and not in the later spec file as well as a much lower priority. And as the bash is part of the initial setup of the build system and all other packages becomes removed at build time the link /usr/bin/sh will point always to /usr/bin/bash (over /etc/alternatives/sh) Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
Thorsten Kukuk <kukuk@suse.de> wrote:
The problem is the installation dependency cycle: - bash requires update-alternatve - update-alternative requires bash
If you install bash first, the update-alternative call in the %post section will fail and there is no /bin/sh.
This is not a problem as long as you install bash as /bin/bash and include bash in a packet that does not depend on things that bas does not need to depend on.
What I'm still missing is: what is the advantage to be able to replace /bin/sh with something different than bash? bash will be installed anyways, there is no way to install an openSUSE without bash, there are too many core packages requiring /bin/bash.
Other shells are e.g. faster and this could speed up the boostrap process. bosh is typically 20% faster than bash and still 4% faster than dash even though bosh implements the POSIX required support for multi byte characters. In general, it is a nice idea to prevent people from introducing constructs into a "#!/bin/sh script" that are not granted to be in a shell since in general it is no longer a correct assumption to have bash in /bin/sh on Linux. In other words: performance and portability is enhanced by such a decision. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 14.03.19 um 07:17 schrieb Dr. Werner Fink:
It should be mentioned that with e.g.
%post -p <lua>
the os.execute() and io.popen() lua API do use /bin/sh just like (g)libc system(3), that is ther is no os.execve() :P
posix.exec is added by rpm Greetings, Stephan -- Lighten up, just enjoy life, smile more, laugh more, and don't get so worked up about things. Kenneth Branagh -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
"Dr. Werner Fink" <werner@suse.de> wrote:
I vote for configure the system Bourne SHell on the command line. As an example I provide the new package posh (Policy-compliant Ordinary SHell) in the project shells. With this the System Administrator has to do
If you start with this decision, you will fail with a high probability because posh is known to be full of bugs. There are better alternatives for an introducing test. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 14, 2019 at 10:44:01AM +0100, Stephan Kulow wrote:
Am 14.03.19 um 07:17 schrieb Dr. Werner Fink:
It should be mentioned that with e.g.
%post -p <lua>
the os.execute() and io.popen() lua API do use /bin/sh just like (g)libc system(3), that is ther is no os.execve() :P
posix.exec is added by rpm
Woh ... luxus pur :D Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
On Thu, Mar 14, 2019 at 10:49:37AM +0100, Joerg Schilling wrote:
"Dr. Werner Fink" <werner@suse.de> wrote:
I vote for configure the system Bourne SHell on the command line. As an example I provide the new package posh (Policy-compliant Ordinary SHell) in the project shells. With this the System Administrator has to do
If you start with this decision, you will fail with a high probability because posh is known to be full of bugs. There are better alternatives for an introducing test.
Which software is free of bugs ... OK, the final TeX/Web from Donald E. Knuth. The only known I see is the the well defined bugs of the posh: Legitimate bugs are inconsistencies between manpage and behavior, and inconsistencies between behavior and Debian policy (currently SUSv3 compliance with the following exceptions: echo -n, binary -a and -o to test, local scoping). which is OK IMHO :) For a test the build system and the bells and whistles therein the posh is simply perfect (IMHO). Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
On Thursday 2019-03-14 08:13, Ludwig Nussel wrote:
If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage.
Which is exactly what I've said elsewhere; comment 903827 within submit request 680550. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On czw, mar 14, 2019 at 8:13 AM, Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Stephan Kulow schrieb:
On 3/13/19 1:44 PM, Simon Lees wrote:
Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.
It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.
Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.
If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage. On package level we have better options to control what goes where and what gets installed by default, eg by means of Requires in patterns or NON_FTP_PACKAGES.
I feel like it's time to draw a parallel between this and xdm/dm situation, where everybody has xdm installed for no particular reason :P LCP [Stasiek] https://lcp.world -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wednesday 2019-03-13 14:54, Marcus Hüwe wrote:
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.
You can specify it via the %__spec_build_cmd macro. For instance, you could add a
%define __spec_build_cmd /bin/bash -e
line to your spec file.
That looks really insance. Why not just do what specfiles already do? %define _buildshell /bin/bash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 14 Mar 2019 08:41:24 +0100 Dominique Leuenberger / DimStar <dimstar@opensuse.org> wrote:
On Wed, 2019-03-13 at 18:43 +0100, Michal Suchánek wrote:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a
%define __spec_build_cmd /bin/bash -e
Yes, that's helpful.
If we get these alternatives I will add it to the compatibility cruft block together with the %license handling.
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
The vast majoriti of packages uses "configure, make, makeinstall", "%cmake, %cmake_build,, %cmake_install"…
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Just saying - Having a way out is good, opposing it for the sake of opposing it is unfriendly in a community.
Cheers, Dominique
PS: the official way of building packages for openSUSE and SLE will stay with osc build and OBS - And of course, should anybody ever have the idea of switching the defaulit shell in this setup, you can be sure that this won't happen without a staging pass.
If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore. And yes, I do not care for most packages that don't contain any convoluted code. Once you add a cycle you may run into variable scoping differences between shells, though. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2019-03-14 12:21, Michal Suchánek wrote:
If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore.
See the build log? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2019-03-14 10:50, Dr. Werner Fink wrote:
On Thu, Mar 14, 2019 at 10:44:01AM +0100, Stephan Kulow wrote:
Am 14.03.19 um 07:17 schrieb Dr. Werner Fink:
It should be mentioned that with e.g.
%post -p <lua>
the os.execute() and io.popen() lua API do use /bin/sh just like (g)libc system(3), that is ther is no os.execve() :P
posix.exec is added by rpm
Gold bathroom: rpm.execute() is added by rpm. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 14 Mar 2019 12:25:03 +0100 (CET) Jan Engelhardt <jengelh@inai.de> wrote:
On Thursday 2019-03-14 12:21, Michal Suchánek wrote:
If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore.
See the build log?
It's always /bin/sh. It does not show what it is linked to. Thanks MIchal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 3/14/19 12:46 PM, Stefan Seyfried wrote:
Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P
Yast team has experience in converting strange languages to each other - they can cook something! :) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2019-03-14 12:46, Stefan Seyfried wrote:
Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P
Now there is an interesting thing to be said about sh. All the interpreters that sought to replace it quite utterly failed to do so — like awk, perl, python, or whatever — and I think it has to do with the fact that commands are no longer first-class citizens in those replacements. One has to write, at the bare minimum, something like forkandexecp("ls", "-l"), wrapping the entire command in a string and passing it to a call. That's a turn-off given >90% of a sh script is just calling other commands. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 14 Mar 2019 12:46:06 +0100 Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P
How much time is spent executing the spec files, anyway? Who would write those lua scripts? There is quite a disproportion between the number of people who understand shell and the number of people who understand lua. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2019-03-14 13:12, Michal Suchánek wrote:
On Thu, 14 Mar 2019 12:46:06 +0100 Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:
The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.
Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P
How much time is spent executing the spec files, anyway?
Who would write those lua scripts? There is quite a disproportion between the number of people who understand shell and the number of people who understand lua.
I can concur: shell has so many oddities over lua (think = vs == and [ vs [[) and people don't seem to understand them, it's why they mix it up in e.g. the build recipes. ;-) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello early Friday on Thursday afternoon! On Mar 14 12:46 Stefan Seyfried wrote (excerpt):
Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P
We need a run-time compiled thingy to get it up to speed. Kind Regards Johannes Meixner -- SUSE LINUX GmbH - GF: Felix Imendoerffer, Jane Smithard, Graham Norton - HRB 21284 (AG Nuernberg) PS: ;-) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
"Dr. Werner Fink" <werner@suse.de> wrote:
Which software is free of bugs ... OK, the final TeX/Web from Donald E. Knuth. The only known I see is the the well defined bugs of the posh:
Legitimate bugs are inconsistencies between manpage and behavior, and inconsistencies between behavior and Debian policy (currently SUSv3 compliance with the following exceptions: echo -n, binary -a and -o to test, local scoping).
which is OK IMHO :)
Do you believe, software generally documents all it's bugs in the man page? BTW: Even if you ignore the bugs related to e.g. "command" and shell arithmetik, posh is approx. 40% slower than pbosh or ksh93 and still 22% slower than bash if you use it to run "configure" fom gtar. An important fact here is that posh does not have a builtin "printf". Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 14.03.19 um 13:12 schrieb Michal Suchánek:
How much time is spent executing the spec files, anyway?
Who would write those lua scripts?
I, for "my" packages. I'm not proposing to do this for all packages. I'm just interested if there would be objections if I converted all the packages that I maintain to use lua instead of shell.
There is quite a disproportion between the number of people who understand shell and the number of people who understand lua.
Oh! A second advantage. No more fiddling around in "my" specfiles by people running "spec-cleaner" or other crap. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 14, 2019 at 01:52:48PM +0100, Joerg Schilling wrote:
"Dr. Werner Fink" <werner@suse.de> wrote:
Which software is free of bugs ... OK, the final TeX/Web from Donald E. Knuth. The only known I see is the the well defined bugs of the posh:
Legitimate bugs are inconsistencies between manpage and behavior, and inconsistencies between behavior and Debian policy (currently SUSv3 compliance with the following exceptions: echo -n, binary -a and -o to test, local scoping).
which is OK IMHO :)
Do you believe, software generally documents all it's bugs in the man page?
BTW: Even if you ignore the bugs related to e.g. "command" and shell arithmetik, posh is approx. 40% slower than pbosh or ksh93 and still 22% slower than bash if you use it to run "configure" fom gtar.
An important fact here is that posh does not have a builtin "printf".
As already said: it is a proof of concept if the build system can handle this. It can and even if installed in a regular system it works as long as the rpm does not include /usr/bin/sh in the file list. Beside this speed is not all (IMHO) ... yes ksh93 was/is fast, but the bug reports had been an annoying experience Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
Jan Engelhardt schrieb:
On Thursday 2019-03-14 08:13, Ludwig Nussel wrote:
If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage.
Which is exactly what I've said elsewhere; comment 903827 within submit request 680550.
Ah I think there is a misunderstanding. The matter probably mixes the unfinished usr move with the /bin/sh alternatives topic. Assuming the usr move is done we are talking about alternatives to /usr/bin/sh. So bash would have to contain /usr/bin/bash and a subpackage of bash the link /usr/bin/sh -> bash. What to do with /bin/{ba,}sh would be a topic for the usr move. Following the current way other packages do it they'd have to statically point to their counterpart in /usr then and be packaged in the same package as the /usr version (see also boo#1029961). cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2019-03-14 12:17:25 +0100, Jan Engelhardt wrote:
On Wednesday 2019-03-13 14:54, Marcus Hüwe wrote:
It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.
You can specify it via the %__spec_build_cmd macro. For instance, you could add a
%define __spec_build_cmd /bin/bash -e
line to your spec file.
That looks really insance. Why not just do what specfiles already do?
%define _buildshell /bin/bash
That would also influence %prep, %install etc. I just realized that __spec_build_shell might be the better choice if one wants to change build shell only. However, in all cases (%_buildshell, %__spec_build_cmd, %__spec_build_shell), one should also consider the __spec_build_template and related macros... Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2019-03-14 12:29, Michal Suchánek wrote:
On Thu, 14 Mar 2019 12:25:03 +0100 (CET) Jan Engelhardt <jengelh@inai.de> wrote:
On Thursday 2019-03-14 12:21, Michal Suchánek wrote:
If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore.
See the build log?
It's always /bin/sh. It does not show what it is linked to.
If and when sh is under u-a control, u-a should be emitting something like Setting up ... in auto mode (or similar) when u-a is run as part of %post. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Mar 21, 2019 at 01:44:08AM +0100, Jan Engelhardt wrote:
On Thursday 2019-03-14 12:29, Michal Suchánek wrote:
On Thu, 14 Mar 2019 12:25:03 +0100 (CET) Jan Engelhardt <jengelh@inai.de> wrote:
On Thursday 2019-03-14 12:21, Michal Suchánek wrote:
If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore.
See the build log?
It's always /bin/sh. It does not show what it is linked to.
If and when sh is under u-a control, u-a should be emitting something like
Setting up ... in auto mode
(or similar) when u-a is run as part of %post.
Except shell is special and u-a is not used. Thanks Michal
On Wed, 2022-03-16 at 14:38 +0100, Michal Suchánek wrote:
If and when sh is under u-a control, u-a should be emitting something like
Setting up ... in auto mode
(or similar) when u-a is run as part of %post.
Except shell is special and u-a is not used.
minor correction: … not used *anymore* in tumbleweed (SLE15SP4 and Leap 15.4 will have the u-a based approach) this was actually only changed back in * Fri Oct 22 2021 by Stefan Schubert - Using package bash-sh instead of the update-alternative mechanism. But we still have multiple xxx-sh packages that are mutually exclusive and that install the /bin/sh -> /bin/xxx symlink as needed (currently we have bash-sh, dash-sh, and busybox-sh) Cheers, Dominique
On Thu, Mar 14, 2019 at 08:41:24AM +0100, Dominique Leuenberger / DimStar wrote:
On Wed, 2019-03-13 at 18:43 +0100, Michal Suchánek wrote:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a
%define __spec_build_cmd /bin/bash -e
Yes, that's helpful.
If we get these alternatives I will add it to the compatibility cruft block together with the %license handling.
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
So now I have recieved this patch for kernel: commit aaf726d5cc067139fd2ade9d4fc1ab1c3e4a268d (origin/users/dmueller/packaging/for-next) Author: Dirk Müller <dmueller@suse.com> Date: Wed Mar 9 21:45:15 2022 +0100 - rpm/*.spec.in.in: Avoid bashism in subshells (bsc#1195391) Maybe it's time for %define __spec_build_cmd /bin/bash -e Maybe that is not sufficient because there is also shell used for %(). Thanks Michal
Moin, Am Mittwoch, 16. März 2022, 14:44:15 CET schrieb Dominique Leuenberger / DimStar:
On Wed, 2022-03-16 at 14:38 +0100, Michal Suchánek wrote:
If and when sh is under u-a control, u-a should be emitting something like
Setting up ... in auto mode
(or similar) when u-a is run as part of %post.
Except shell is special and u-a is not used.
minor correction: … not used *anymore* in tumbleweed (SLE15SP4 and Leap 15.4 will have the u-a based approach)
That's going to be reverted, 15SP4/15.4 are on track to use bash-sh like TW. Cheers, Fabian
this was actually only changed back in * Fri Oct 22 2021 by Stefan Schubert - Using package bash-sh instead of the update-alternative mechanism.
But we still have multiple xxx-sh packages that are mutually exclusive and that install the /bin/sh -> /bin/xxx symlink as needed (currently we have bash-sh, dash-sh, and busybox-sh)
Cheers, Dominique
Hello, On Wed, Mar 16, 2022 at 03:05:26PM +0100, Fabian Vogt wrote:
Moin,
Am Mittwoch, 16. März 2022, 14:44:15 CET schrieb Dominique Leuenberger / DimStar:
On Wed, 2022-03-16 at 14:38 +0100, Michal Suchánek wrote:
If and when sh is under u-a control, u-a should be emitting something like
Setting up ... in auto mode
(or similar) when u-a is run as part of %post.
Except shell is special and u-a is not used.
minor correction: … not used *anymore* in tumbleweed (SLE15SP4 and Leap 15.4 will have the u-a based approach)
That's going to be reverted, 15SP4/15.4 are on track to use bash-sh like TW.
And are there u-a based alternatives before that or was it always hardcodedto bash? Thanks Michal
Moin, Am Mittwoch, 16. März 2022, 15:19:12 CET schrieb Michal Suchánek:
Hello,
On Wed, Mar 16, 2022 at 03:05:26PM +0100, Fabian Vogt wrote:
Moin,
Am Mittwoch, 16. März 2022, 14:44:15 CET schrieb Dominique Leuenberger / DimStar:
On Wed, 2022-03-16 at 14:38 +0100, Michal Suchánek wrote:
If and when sh is under u-a control, u-a should be emitting something like
Setting up ... in auto mode
(or similar) when u-a is run as part of %post.
Except shell is special and u-a is not used.
minor correction: … not used *anymore* in tumbleweed (SLE15SP4 and Leap 15.4 will have the u-a based approach)
That's going to be reverted, 15SP4/15.4 are on track to use bash-sh like TW.
And are there u-a based alternatives before that or was it always hardcodedto bash?
FWICT SLE 15 SP3 and below had bash providing /bin/sh. Cheers, Fabian
Thanks
Michal
Hi Michal, Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek <msuchanek@suse.de>:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a %define __spec_build_cmd /bin/bash -e
This is actually pullrequested already for a while, see here: https://github.com/openSUSE/rpm-config-SUSE/pull/53 you don't need to worry about not using bash as the build shell. there are two cases where it could change: * when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.
So now I have recieved this patch for kernel:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range. Thanks, Dirk
On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:
Hi Michal,
Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek <msuchanek@suse.de>:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a %define __spec_build_cmd /bin/bash -e
This is actually pullrequested already for a while, see here:
https://github.com/openSUSE/rpm-config-SUSE/pull/53
you don't need to worry about not using bash as the build shell. there are two cases where it could change:
* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.
So now I have recieved this patch for kernel:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.
And rather than making the code more complex I would opt for BuildRequires: bash-sh that way there will be no surprises even for %(). While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable. Thanks Michal
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range. And rather than making the code more complex I would opt for
Is the code really more complex by this change? -%(chmod +x %_sourcedir/{@SCRIPTS@}) +%(cd %_sourcedir && chmod +x @SCRIPTS@) That's the bulk of the changes. I think the simpler chmod statement makes that one actually a tad easier to read, as you don't have to carefully parse brackets inside parentheses to understand what is going on. The other one is using unquoted " within " which is even recommended against in bash, where a single quote would work with any shell. I can also carry this as a local patch, but it is so minor that I don't think it adds any significant complexity. we're talking about 4 additional characters that, depending on how you see it, remove a {} construct that might be less obvious to understand also.
BuildRequires: bash-sh
that way there will be no surprises even for %().
that would make the kernel unresolvable and I'd have to carry a local patch to revert that so it would only make my work even harder than it already is. Also this would only work within the build service and would not make anything different for anyone using any other method of building the rpms.
While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system.
I agree with that. In order to test that it is installable on a minimal system in the buildservice, I need to have /bin/sh that way, because packages are installed at the end of the build and have their post install scripts executed. I'm near the bottom of this effort, the kernel is one of the last packages that still fails. It works for almost everything (I have about 10 failures including the kernel packages left of the ~ 3000 packages in the ring). I am not patching out complex and useful bash features here. These are hair sized changes that do not harm the understandability of the code. I'm not asking for a lot here, and while I do have a workaround of carrying a local patch and keeping the kernel that way, that costs me significantly more continuous work than upstreaming the change. And the change is not making your hard maintenance work any harder. you don't have to keep this working at all, and the lines I modified haven'T been touched in the last 11 years in git so they're certainly also not a big maintenance complexity for anyone else.
fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.
I can submit a more minimal patch that is 6 lines change in total, no move of code tomorrow. I'm currently testing it. it will not add a single line of code, it will not move or remove a single line of code. it changes 4 characters in total per spec file. Thanks, Dirk
On 3/18/22 06:51, Michal Suchánek wrote:
On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:
Hi Michal,
Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek <msuchanek@suse.de>:
You can specify it via the %__spec_build_cmd macro. For instance, you could
add a %define __spec_build_cmd /bin/bash -e
This is actually pullrequested already for a while, see here:
https://github.com/openSUSE/rpm-config-SUSE/pull/53
you don't need to worry about not using bash as the build shell. there are two cases where it could change:
* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.
So now I have recieved this patch for kernel:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.
And rather than making the code more complex I would opt for
BuildRequires: bash-sh
that way there will be no surprises even for %().
While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.
As someone who cares about the build time of packages like the kernel I would be interested in looking at the differences in build time between using bash and dash as /bin/sh my suspicion is that the kernel build process still launches /bin/sh enough times that using a lighter alternative might make a difference. So if its not adding massive complexity i'd also appreciate keeping support for multiple /bin/sh here atleast until we can do better analysis on it. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Hello On Fri, Mar 18, 2022 at 10:14:30PM +1030, Simon Lees wrote:
On 3/18/22 06:51, Michal Suchánek wrote:
On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:
Hi Michal,
Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek <msuchanek@suse.de>:
You can specify it via the %__spec_build_cmd macro. For instance, you could > add a > %define __spec_build_cmd /bin/bash -e
This is actually pullrequested already for a while, see here:
https://github.com/openSUSE/rpm-config-SUSE/pull/53
you don't need to worry about not using bash as the build shell. there are two cases where it could change:
* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.
So now I have recieved this patch for kernel:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.
And rather than making the code more complex I would opt for
BuildRequires: bash-sh
that way there will be no surprises even for %().
While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.
As someone who cares about the build time of packages like the kernel I would be interested in looking at the differences in build time between using bash and dash as /bin/sh my suspicion is that the kernel build process still launches /bin/sh enough times that using a lighter alternative might make a difference. So if its not adding massive complexity i'd also appreciate keeping support for multiple /bin/sh here atleast until we can do better analysis on it.
Until better analysis is done I see this as a case of premature optimization. My suspicion is that the runtime of the scripts that are not run by bash explicitly is irrelevant. Also there is no intention to change the default shell in the distributions so any savings by lighter shell will be obtained only in test projects. Thanks Michal
Hello, On Thu, Mar 17, 2022 at 10:06:09PM +0100, Dirk Müller wrote:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range. And rather than making the code more complex I would opt for
Is the code really more complex by this change?
-%(chmod +x %_sourcedir/{@SCRIPTS@}) +%(cd %_sourcedir && chmod +x @SCRIPTS@)
slightly
That's the bulk of the changes. I think the simpler chmod statement makes that one actually a tad easier to read, as you don't have to carefully parse brackets inside parentheses to understand what is going on.
That's completely subjective
The other one is using unquoted " within " which is even recommended against in bash, where a single quote would work with any shell.
I am not against improving readability of shell code in general, and this may fall in that category
I can also carry this as a local patch, but it is so minor that I don't think it adds any significant complexity. we're talking about 4 additional characters that, depending on how you see it, remove a {} construct that might be less obvious to understand also.
BuildRequires: bash-sh
that way there will be no surprises even for %().
that would make the kernel unresolvable and I'd have to carry a local patch to revert that so it would only make my work even harder than it already is. Also this would only work within the build service and would not make anything different for anyone using any other method of building the rpms.
So long as you build rpms the dependencies should be fulfilled so it will work for any method that builds rpms. If you don't build rpms you supply your own build scripts and you are free to write them in any language you choose.
While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system.
I agree with that. In order to test that it is installable on a minimal system in the buildservice, I need to have /bin/sh that way,
Buildservice is a build environment, not a test environment. By testing in buildservice you are conflating build time and runtime. Please use the correct tool for the job. It's not like we don't have any.
because packages are installed at the end of the build and have their post install scripts executed.
I'm near the bottom of this effort, the kernel is one of the last packages that still fails. It works for almost everything (I have about 10 failures including the kernel packages left of the ~ 3000 packages in the ring).
I suppose those 10 are those that actually use non-trivial scripting during the build.
I am not patching out complex and useful bash features here. These are hair sized changes that do not harm the understandability of the code. I'm not asking for a lot here, and while I do have a workaround of carrying a local patch and keeping the kernel that way, that costs me significantly more continuous work than upstreaming the change. And the change is not making your hard maintenance work any harder. you don't have to keep this working at all, and the lines I modified haven'T been touched in the last 11 years in git so they're certainly also not a big maintenance complexity for anyone else.
And when one of the alternative shells subtly breaks some of the code that is not explicitly executed in bash because there is no way to specify the interpreter for it will you also request that the kernel carries compatibility cruft for such breakage? Where do you draw the line? I don't support building with anything but bash, to reduce maintenance effort. Now that the distribution grew /bin/sh alternatives I will request that for kernel build /bin/sh is bash, always. Thanks Michal
On 3/19/22 03:15, Michal Suchánek wrote:
Hello
On Fri, Mar 18, 2022 at 10:14:30PM +1030, Simon Lees wrote:
On 3/18/22 06:51, Michal Suchánek wrote:
On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:
Hi Michal,
Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek <msuchanek@suse.de>:
> You can specify it via the %__spec_build_cmd macro. For instance, you could >> add a >> %define __spec_build_cmd /bin/bash -e
This is actually pullrequested already for a while, see here:
https://github.com/openSUSE/rpm-config-SUSE/pull/53
you don't need to worry about not using bash as the build shell. there are two cases where it could change:
* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()
And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.
There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.
So now I have recieved this patch for kernel:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.
And rather than making the code more complex I would opt for
BuildRequires: bash-sh
that way there will be no surprises even for %().
While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.
As someone who cares about the build time of packages like the kernel I would be interested in looking at the differences in build time between using bash and dash as /bin/sh my suspicion is that the kernel build process still launches /bin/sh enough times that using a lighter alternative might make a difference. So if its not adding massive complexity i'd also appreciate keeping support for multiple /bin/sh here atleast until we can do better analysis on it.
Until better analysis is done I see this as a case of premature optimization. My suspicion is that the runtime of the scripts that are not run by bash explicitly is irrelevant.
Well Dirk's changes will allow us to properly analyse this.
Also there is no intention to change the default shell in the distributions so any savings by lighter shell will be obtained only in test projects.
One of the advantages of this change is it doesn't need a change to the distro default shell for us to benefit from it. If there is a benefit to kernel build times we can simply add BuildRequires: dash-sh to kernel packagse. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Sat, Mar 19, 2022 at 03:03:47PM +1030, Simon Lees wrote:
On 3/19/22 03:15, Michal Suchánek wrote:
Hello
On Fri, Mar 18, 2022 at 10:14:30PM +1030, Simon Lees wrote:
On 3/18/22 06:51, Michal Suchánek wrote:
On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:
Hi Michal,
Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek <msuchanek@suse.de>:
>> You can specify it via the %__spec_build_cmd macro. For instance, you could >>> add a >>> %define __spec_build_cmd /bin/bash -e
This is actually pullrequested already for a while, see here:
https://github.com/openSUSE/rpm-config-SUSE/pull/53
you don't need to worry about not using bash as the build shell. there are two cases where it could change:
* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()
> And I'm sure the review team will decline them if there is no proper > reason to do this - if you don't have at least *some* complexity in the > build section.
There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.
So now I have recieved this patch for kernel:
Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.
And rather than making the code more complex I would opt for
BuildRequires: bash-sh
that way there will be no surprises even for %().
While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.
As someone who cares about the build time of packages like the kernel I would be interested in looking at the differences in build time between using bash and dash as /bin/sh my suspicion is that the kernel build process still launches /bin/sh enough times that using a lighter alternative might make a difference. So if its not adding massive complexity i'd also appreciate keeping support for multiple /bin/sh here atleast until we can do better analysis on it.
Until better analysis is done I see this as a case of premature optimization. My suspicion is that the runtime of the scripts that are not run by bash explicitly is irrelevant.
Well Dirk's changes will allow us to properly analyse this.
Will it? The kernel is built on different builder each time with build times varying wildly. Running a build with different shell once or twice gives absolutely no insight into the impact of using different shells. Thanks Michal
On Saturday 2022-03-19 13:39, Michal Suchánek wrote:
Well Dirk's changes will allow us to properly analyse this.
Will it?
The kernel is built on different builder each time with build times varying wildly. Running a build with different shell once or twice gives absolutely no insight into the impact of using different shells.
Add a _constraint that limits it to a particular worker.
On 3/20/22 19:57, Jan Engelhardt wrote:
On Saturday 2022-03-19 13:39, Michal Suchánek wrote:
Well Dirk's changes will allow us to properly analyse this.
Will it?
The kernel is built on different builder each time with build times varying wildly. Running a build with different shell once or twice gives absolutely no insight into the impact of using different shells.
Add a _constraint that limits it to a particular worker.
Yeah or what we generally do is build multiple times for each configuration and then we can work out what the difference is on both the slower and faster workers. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
participants (25)
-
Andreas Schwab -
Carlos E. R. -
Dirk Müller -
Dominique Leuenberger / DimStar -
Dr. Werner Fink -
Fabian Vogt -
H.Merijn Brand -
Jan Engelhardt -
Joerg Schilling -
Johannes Meixner -
Knurpht-openSUSE -
Ludwig Nussel -
Marcus Hüwe -
Martin Wilck -
Michal Kubecek -
Michal Suchánek -
Mischa Salle -
Neal Gompa -
Patrick Shanahan -
Richard Brown -
Simon Lees -
Stasiek Michalski -
Stefan Seyfried -
Stephan Kulow -
Thorsten Kukuk