On Friday 2019-02-08 15:03, Martin Wilck wrote:

On Fri, 2019-02-08 at 14:25 +0100, Jan Engelhardt wrote:

...

On Friday 2019-02-08 13:22, Martin Wilck wrote:

...

I'm aware that, in principle, /bin/sh is supposed to be the Bourne shell on Unix systems. But as a matter of fact, on current openSUSE, it is not.

But it could be. In fact, it can be. `zypper dup` yesterday with an update block of 120 packages yielded only 7 scriptlet problems when sh is dash.

For me the question is, what's the benefit?

Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.

...

No one needs to write sh-compatible code. The code just needs to be annotated that it requires a certain interpreter. Since script files already have a #! line most of the time, that's taken care of.

... unless that line reads '#! /bin/sh'.

Debian is one distro where dash is prominently featured in one capacity or another, so upstream software already has to deal with the situation. Which leaves us with SUSE-specific software, and from a glance at e.g. github.com/openSUSE/sysconfig, they all quite consistently declare bash. I would argue that, in fact, people rather write #!/bin/bash these days just because it's in more or less every howto, and no newbie wants to have to deal the significant changes that a POSIX sh would bring.

...

Since I literally get to see every new line of spec code due to factory-review, many specfiles have already gotten their treatment (if one was even needed). The upside is that specfiles don't need a lot of work because everyone tries to keep them short anyway and reuse macros due to the volume of it all.

What rules do you follow in this review work?

* Does it build? No => back to the drawing board (reject) * https://en.opensuse.org/openSUSE:Creating_a_changes_file_(RPM) fulfilled? No => give up, complain (comment), or let someone else have the honor to reject it * https://en.opensuse.org/openSUSE:Package_description_guidelines fulfilled? No => complain or outright make an independent SR to the develprj/DP for evaluation and debate * https://en.opensuse.org/openSUSE:Packaging_guidelines#Licensing https://en.opensuse.org/openSUSE:Packaging_guidelines#Legal (only sometimes and only cursory, since there's a dedicated legal department and robots doing that already) metadata mismatch => reject * https://en.opensuse.org/openSUSE:Trademark_guidelines failure to abide => SR to DP (the bots don't catch it ATM) * https://en.opensuse.org/openSUSE:Packaging_guidelines and https://en.opensuse.org/openSUSE:Specfile_guidelines nonconformance => SR to DP, so I get the credit for fixing it * Anything else appears worth fixing => SR to DP

Do you require POSIX compliance?

No.

Or compliance with dash's features, which is not the same?

No.

Do other reviewers follow the same rules as you do?

Reviewers may forget some points sometimes letting bad stuff pass through, but they be forgiven for the guidelines are so plentiful and humans have a certain MTBF. So... yes, they follow the "rules". Do they follow the same algorithm? I suspect not everyone will jump in and proactively fix a submission for the submitter, because that constitutes a development not a review activity. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Wilck wrote:

On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:

...

On Friday 2019-02-08 15:03, Martin Wilck wrote:

...

For me the question is, what's the benefit?

Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.

That's truly interesting. But I can bounce back your argument: you could surely run "pbosh ./configure" to the same effect (what's pbosh, btw? did you mean posh?).

"posh" is not a shell but a joke. It is full of bugs and cannot be seen as a serious POSIX shell implementation. "pbosh" is a compile variant of "bosh" which is the maintained portable and enhanced Bourne Shell that allows to be compiled compatible to the SVr4 Bourne Shell (obosh), a minimal strict POSIX variant (pbosh) and an enhanced POSIX variant (bosh). People who like to run "configure" tests as benchmarks should know that there is a need to make sure that sub-shells are called correctly. You need to call something like: CONFIG_SHELL=$shell $shell ./configure

Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.

This is not unfair. If you like to compare, I recommend you to use a ksh93 from 2008 which is the fastest shell ever and edit "configure" to switch off the important related ksh93 builtins. Note that ksh93 from 2008 is nearly twice as fast as bash. The most called programs are: rm cat sed tr mv in the order of usage. The reason why ksh93 is so fast is: - it uses vfork(), this is also true for bosh since 4 years. - it has a rewritten parser/interpreter that is larger (more code) but faster than the historic Bourne Shell This also applies to bosh - ksh93 does not fork for "(subcmd)". It rather creates a new set of "global variables" in a structure. This is ksh93 specific. - ksh93 uses an own i18n implementation with interface functions that better fit to the needs of a shell.

There is some evidence that bash is not the fastest of shells ( https://unix.stackexchange.com/questions/148035/is-dash-or-some-other-shell-... ). OTOH, shell script performance matters much less these days than it used to.

dash is approx. 30% faster than bash because it does not suport multi-byte characters.

But isn't that also an argument for having /bin/sh always point to bash? Like it or not, bash is the "de-facto standard" (yuck) shell today under Linux. People expect it to be that way.

BTW: Most of the bash "extensions" are taken from ksh93. The POSIX standard shell definition was derived from the features from ksh88 with [[ ]], (( )) and array variables removed. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Mon, Feb 11, 2019 at 10:21:47AM +0100, Martin Wilck wrote:

Speaking about shell script performance, IMO the only set of scripts for which this matters in practice is dracut, which would benefit a LOT from 30% speedup. Unfortunately it was written as a "#! /bin/bash" script in the first place.

Indeed, nevertheless as dracut uses bashism it should enforce to use /bin/bash, otherwise arrays, types, [[..]], ((..)), ... had been rewritten and maybe other external tools must be added as replacements. Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr

On Mon, 11 Feb 2019 21:16:36 +1030 Simon Lees wrote:

On 11/02/2019 19:51, Martin Wilck wrote:

...

On Fri, 2019-02-08 at 19:07 +0100, Joerg Schilling wrote:

...

Martin Wilck wrote:

...

On Fri, 2019-02-08 at 17:35 +0100, Jan Engelhardt wrote:

...

On Friday 2019-02-08 15:03, Martin Wilck wrote:

...

For me the question is, what's the benefit?

Off the top of my head and not very scientifically measured, ./configure for kopano.spec completes 15% faster with pbosh.

People who like to run "configure" tests as benchmarks should know that there is a need to make sure that sub-shells are called correctly.

You need to call something like:

CONFIG_SHELL=$shell $shell ./configure

...

Btw the comparison is a bit unfair: "configure", by design, is written in the most portable shell dialect possible, forfeiting all chances for bash to catch up with optimized builtins.

This is not unfair. If you like to compare, I recommend you to use a ksh93 from 2008 which is the fastest shell ever and edit "configure" to switch off the important related ksh93 builtins. Note that ksh93 from 2008 is nearly twice as fast as bash.

Speaking about shell script performance, IMO the only set of scripts for which this matters in practice is dracut, which would benefit a LOT from 30% speedup. Unfortunately it was written as a "#! /bin/bash" script in the first place.

Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.

And that is the question. Will it make any difference? Many packages that build slow spend a lot of time actually *compiling* stuff. Many packages that build slow spend a lot of time in rpm checkers. At least in one case it has been shown that rpm checkers would benefit from using all of bash features such as [[]] because it has much common functionality built-in and much time is spent forking off tools to do small tasks that a POSIX-compliant shell cannot do on its own but bash can. So which are the packages that actually do benefit from faster bare-bones /bin/sh? Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Mon, 11 Feb 2019 12:18:58 +0100 (CET) Jan Engelhardt wrote:

On Monday 2019-02-11 12:07, Michal Suchánek wrote:

...

...

Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.

Many packages that build slow spend a lot of time actually *compiling* stuff.

Yes and no. Build times follow the usual long-tail distribution (Pareto or something). Hugeass packages like chromium of course spent 99% their time in g++, while things like python-* are probably spending most of their time in /usr/bin/build and /usr/bin/rpm and disk IO. And we don't exactly have few packages in the {python, perl, ruby} camp, mind you.

...

Many packages that build slow spend a lot of time in rpm checkers.

But for different reasons. * The compression time scales linearly with the size of the contents. * absolute executable size roughly scales linearly with package size (something in the ballpark of 60% of a package are executables normally; ignoring noarch packages) * ELF analysis (dwz et al) scales linearly with executable size * the size of executable scales linearly with project size

Which is the reason things like chromium require exorbitant time and are very frustrating if it happens to fail at the last moment, while packages like cdecl are comparatively trivial - but have a high buildroot setup cost.

And you forgot to quote the important part: is the setup cost mitigated by shell scripts that use fast bare-bones shell and have to fork() for every tiny task or shell scripts that fully use bash builtins? Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 11/02/2019 21:48, Jan Engelhardt wrote:

On Monday 2019-02-11 12:07, Michal Suchánek wrote:

...

...

Looking slightly more broadly if changing #!/bin/sh to something lighter lead to a 10-20% reduction in build times for some of our slower building packages, that is something that would interest my team and I guess should also interest the obs team and anyone who cares about our infrastructure usage.

Many packages that build slow spend a lot of time actually *compiling* stuff.

Yes and no. Build times follow the usual long-tail distribution (Pareto or something). Hugeass packages like chromium of course spent 99% their time in g++, while things like python-* are probably spending most of their time in /usr/bin/build and /usr/bin/rpm and disk IO. And we don't exactly have few packages in the {python, perl, ruby} camp, mind you.

...

Many packages that build slow spend a lot of time in rpm checkers.

But for different reasons. * The compression time scales linearly with the size of the contents. * absolute executable size roughly scales linearly with package size (something in the ballpark of 60% of a package are executables normally; ignoring noarch packages) * ELF analysis (dwz et al) scales linearly with executable size * the size of executable scales linearly with project size

Yeah for many larger packages up to 1/3rd of the build time is spent after make install is finished, but for larger slower building packages as mentioned above the issue is not the rpm checkers, for slow, large packages the rpm checkers only make up a very small amount of the build time whereas for small quick packages the rpm checkers take a larger percentage of time which is why we already invested a bit of time in speeding them up. For large packages the time is mostly spent doing compression and extracting the debug symbols, this is currently not done in parallel so we waste huge amounts of CPU time here currently. Performing this step in parallel by processing multiple subpackages at once does provide a significant improvement, however given rpmbuild was never designed to do this in parallel I am yet to be able to do it reliably without crashes due to race conditions accessing memory or deadlocks so i'm not actively working on it atm. Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium. If we can get to a point where we are speeding up these kind of packages by some amount maybe it makes sense to add an update alternatives setup for #!/bin/sh even if its mostly so we can use it in the build root of certain packages to speed them up. At the moment our team is still looking into a bunch of smaller ways to speed up the build in the hope if we find enough it will eventually get to something more substantial, its an area people haven't spent a lot of time looking at so we are still finding some low hanging fruit. From our point of view we are trying to reduce the time it takes to release security updates and one of the ways to do that is obviously speeding up build times. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

On Tue, Feb 12, 2019 at 07:09:18AM +0100, Michal Kubecek wrote:

On Tuesday, 12 February 2019 1:59 Simon Lees wrote:

...

Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at. Particularly in regards to packages like the kernel, libreoffice and chromium.

It certainly can. Few years ago, I was rather frustrated that when building a kernel package locally, more than 6 minutes out of total 35 was spent by brp-symlinks script. I was even considering to rewrite it to C and parallelize as a hackweek project.

Then Tomáš Čech tried, as an exercise, to rewrite the script just by replacing various invokations of sed, grep etc. with bash specific expansions. As a result, he managed to squeeze those 6 minutes down to ~30 seconds.

I can second this. Using bash builtins and string features can speed up scripts a lot. That is avoiding forking often for external command within loops, using <() fifo together with an external command to handle large lists of lines of strings at once and read the resulting lines with loops only using bash builtins. Also ksh93 loops can read from pipe (which is actual a combination of two socketpair()s) plus avoiding subprocess for the loop its self. But this is not portable to many bourne shell scripts as with ksh93 variables within the loop are visiable outside the loop. This can be an advantage but may break scripts depending on orignal bourne shell behaviour. The bash provides the <() fifo as a replacment for this. Then one can use the redirection operator to read from the <() fifo as stdin. -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr

H.Merijn Brand wrote:

Projects that require features - like bash functionality - that is not available on those proprietary OS's will cause the support on those OS's to halt. e.g.

If you depend on proprietary bash features, you are not portable. This is why POSIX exists. I modified a few shell scripts from OpenSolaris to no longer depend on ksh93 to be able to use "bosh" as /bin/sh and /sbin/sh for the basic startup before /usr gets mounted. On the other side: as long as bash stays portable it is not a real problem to use bash as long as you put #!/bin/bash in the first line of your script. I guess this will not change as long as bash stays a one man show with the original maintainer.

? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc

It is bad to see that some GNU projects more and more implement the embrace and extent method from Microcoft to create a vendor lock in instead of using POSIX features that are portable.

I know that OSS developer time is "free" time and voluntary work, but remember that choosing the tool-of-the-day will cause headaches for those that use their free time to port OSS to those OS's that are not mainstream anymore.

This is e.g. why the ksh93 from Redhat no longer compiles on Solaris. The background in this case BTW. is missing skills by the related people.

Story from the past: we had to switch from SCCS to something more recent. The team chose SVN. Our main development environment by the time was HP-UX 11.23, and we had to give up on SVN: we needed a week to build SVN from scratch with all its dependencies only to see it horribly fail. We then decided to try git. We just needed 3 hours to get it up and running, including the GUI's. *I* do not regret this, as I had a preference of git over svn from the start, and by now all our developers are used to git/gitlab and it is part of daily routine.

Well, I could never understand why peole would like to use svn as it is as slow as CVS and as it did even cause some projects to loose all their history after a internal change of the datastructures.

If bash were to switch from make to meson/ninja, you'd rule out the use of modern bash on those weird OS's, as bash still is relative easy to build/port. (and so is "make")

Well writing makefiles that depend on "gmake" is a bad practice, but it is still better than using something else. If you are interested in a portable POSIX certified make program, I recommend to use the "SunPro Make" from schilytools. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 2019-02-13 at 09:50 +1030, Simon Lees wrote:

On 12/02/2019 21:50, Joerg Schilling wrote:

...

H.Merijn Brand wrote:

...

Projects that require features - like bash functionality - that is not available on those proprietary OS's will cause the support on those OS's to halt. e.g.

If you depend on proprietary bash features, you are not portable. This is why POSIX exists. I modified a few shell scripts from OpenSolaris to no longer depend on ksh93 to be able to use "bosh" as /bin/sh and /sbin/sh for the basic startup before /usr gets mounted.

On the other side: as long as bash stays portable it is not a real problem to use bash as long as you put #!/bin/bash in the first line of your script. I guess this will not change as long as bash stays a one man show with the original maintainer.

...

? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc

Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.

Meson being a python3 app would be a terrible choice for anythingn in the distro bootstrap cycle (aka Ring0). That cycle is large enough as it is - pulling python3 into it won''t make it better. So I hope the idea of bash moving to meson is just a random though nobody really considers. Cheers Dominique

On Wednesday, 13 February 2019 0:20 Simon Lees wrote:

Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.

https://twitter.com/mesonbuild/status/1095391825647538176

It's hard to believe just switching to meson could achieve this without something being deeply wrong with the previous build system and that one couldn't improve the build times significantly even without meson. Michal Kubecek -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 13 Feb 2019 09:50:51 +1030, Simon Lees wrote:

...

...

? GNU gcc is by now impossible to build on HP-UX. ? Projects that switch(ed) to meson/ninja won't build outside Linux ? Projects that require new(er/est) gcc won't build on OS's that won't build gcc

Interestingly related to this discussion I read from the Meson twitter account this morning "Switching to Meson made KornShell's build time 35x faster. Not 35 percent, 35 times." which is a pretty compelling argument for making a change.

https://twitter.com/mesonbuild/status/1095391825647538176

Maybe they forgot that make supports -j35 meson is "parallel" by default, make is not And I fully subscribe to Dominique's comment, but that might be obvious -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

Andreas Schwab wrote:

That only tells how terrible the ksh build system was.

It was portable, so it was definitely better that what these people introcuded. So in general, be careful about fake news.... Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

"Dr. Werner Fink" wrote:

I can second this. Using bash builtins and string features can speed up scripts a lot. That is avoiding forking often for external command within loops, using <() fifo together with an external command to handle large lists of lines of strings at once and read the resulting lines with loops only using bash builtins.

In ksh93, <() works using /dev/fd/* The main difference to $(cmd) is that the output of the command is a normal pipe instead of letting the shell read from that pipe and create an argument list from it. For bash that did not yet rewrite it's parser to implement more efficient pipelines, this may give an advantage, but for shells that run the rightmost command in the main shell in case it is a builtin, there is no difference. So this does not give performance benefits in ksh93 and it would not give performance benefits if bosh did start to implement <().

Also ksh93 loops can read from pipe (which is actual a combination of two socketpair()s) plus avoiding subprocess for the loop its self. But this is not portable to many bourne shell scripts as with ksh93 variables within the loop are visiable outside the loop. This can be an advantage but may break scripts depending on orignal bourne shell behaviour. The bash provides the <() fifo as a replacment for this. Then one can use the redirection operator to read from the <() fifo as stdin.

See above. The original Bourne Shell implementation from 1976 had a main problem: program size could not exceed 64k on a PDP11 unless you used slow overlays (as used in vi). So the Bourne Shell has been written in a way that created the smallest program regardless of the speed of that program. ksh93 uses a new method to create pipelines. This is faster and it allows to use vfork() to speed up things. vfork() on a real UNIX system is typically 3-4x faster than fork() as it does not need to copy the address space description. This still applies even to modern UNIX versions like SunOS-4.x that introduced a copy on write fork() in 1988. This is where Linux could speed up BTW...as vfork() is using a copy on write based emulation on top of fork(). To be able to use vfork(), you need to use a different chain of processes when creating the processes for a pipeline. This requires a rewrite of the parser and the interpreter in the shell. Once you did this, it is not a big deal anymore to run the rightmost program of a pipeline in the main shell process in case it is a builtin command. This is what finally allows you to set up shell macros that are seen by the main shell, as in: echo bla | read VAR Another strange thing from the historic Bourne Shell is that a while loop with I/O redirection always has been run in a subshell. Ksh93 changed this. Since bosh also implements both optimizations mentioned above, typical shell scripts like "configure" now run 30% faster. ksh93 uses virtual sub-shells to avoid fork() in many cases and gets another speed up, but the version of ksh93 that is created by RedHat people on github is no longer faster than "bosh" as important code has been removed or destroyed. BTW: if you check the speed of "configure" in special, you will see that "echo" and "test" being a builtin makes the biggest difference and with GNU configure past 2.13, the biggest time consumer is the fact that "printf" is used instead of "echo" and if that is not built into the shell, a configure run on such a shell is extremely slow. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

"Dr. Werner Fink" wrote:

Hmmm ... AFAICR ksh93 was huge collection of bugs causing a lot of bugzilla entries (tagged 56 bugs). I can remember the problems the $(cmd) had caused whereas the old `cmd` code had worked flawless. I had spent a lot of time to hunt bugs in the multibyte code as well. One bug was memory allocing code in the signal handling. During maintaining systemd I had no time anymore to hunt further bugs down hence the new maintainer for ksh93/libast. From my point of view I prefere a stable shell with maintainable code more than a exterm fast shell with bugs and interwoven bad documented code.

Such problems did not exist on Solaris and a non-portable ksh93 kills it. The main advantage from ksh93 was is portability as long as David Korn was the maintainer. I don't like the current version of ksh93 even though it fixed a few bugs. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Tue, 2019-02-12 at 11:29 +1030, Simon Lees wrote:

Yeah for many larger packages up to 1/3rd of the build time is spent after make install is finished,

Out of the checks we run, only "post-build-checks" consists of shell code. rpmlint and rpmlint-mini are python, and brp-check-suse is perl. And "post-build-checks" script use a "bash" shebang. The same holds for the "build" package. So IMO nothing would be gained wrt build checks by changing what /bin/sh points at. It'd make more sense to review "post-build-checks" and "build" for bashisms, and possibly use a different, faster shell for them. That'd be much less review work than doing it for the whole distro.

Before this thread we hadn't considered looking at whether using a different #!/bin/sh would have a impact on builds particularly in the time it takes to execute configure or process makefiles under different shells, at some point in the next couple of months its now on my list to look at.

As mentioned already, for that it'd be sufficient to use a different shell for running "configure" (iow modify the %configure macro). Regards Martin -- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Michal Suchánek wrote:

Many packages that build slow spend a lot of time actually *compiling* stuff.

Are you shure? Before I did run performance analysis on "configure", I thought that the compiler is the most time consuming thing in this area and I failed completely with this assumption. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Mon, 11 Feb 2019 12:40:32 +0100 Martin Wilck wrote:

On Mon, 2019-02-11 at 12:24 +0100, Joerg Schilling wrote:

...

Michal Suchánek wrote:

...

Many packages that build slow spend a lot of time actually *compiling* stuff.

Are you shure?

Before I did run performance analysis on "configure", I thought that the compiler is the most time consuming thing in this area and I failed completely with this assumption.

IIRC even "configure" spends a lot of time compiling, as it forks the compiler many times for testing things. Indeed there are packages where "configure" takes substantially more time that the actual "make".

Anyway, for speeding up "configure", we could simply replace the "./configure" call with

"$CONFIG_SHELL=$someshell $someshell ./configure"

And even have %configure macro do it for you. Thanks Michal

in those spec files where this really matters. So I reckon that "configure" speed is not a strong argument for making /bin/sh customizable.

Martin

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, 2019-02-08 at 20:01 +0100, Jan Engelhardt wrote:

On Friday 2019-02-08 18:22, Martin Wilck wrote:

...

...

But isn't that also an argument for having /bin/sh always point to bash? Like it or not, bash is the "de-facto standard" (yuck) shell today under Linux. People expect it to be that way.

"We always did it that way" is just not a very good argument for anything.

That wasn't my argument. The argument was that changing it would break people's assumptions. But never mind.

...

Back to my point - openSUSE needs to have a reliable standard what kind of shell syntax is acceptable in scriptlets which don't use "- p".

The aim surely is sh, simply because most specfiles are simple, already compliant or annotated -- judging from all the files that have passed me by within the last two-plus years.

Define "sh", please. We've seem at least 4 candidates in this thread so far except bash itself (dash, ash, (p)bosh, ksh93). Martin -- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, 8 Feb 2019 14:25:38 +0100 (CET), Jan Engelhardt wrote:

No one needs to write sh-compatible code.

I do. But not on Linux My .sh scripts also have to run on ancient AIX and HP-UX, and neither comes with bash. Whenever I need more in a shell-script than old bourne syntax, I turn to perl. I /think/ I never wrote a sh/bash script that even uses $() -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

Am 08.02.19 um 18:30 schrieb Martin Wilck:

You seem to be arguing for dash: https://wiki.ubuntu.com/DashAsBinSh. But that article also painfully demonstrates all the stuff people need to avoid if they want to write compliant code.

If we really want a small, fast, good shell, I'd vote for busybox's ash. It supports many useful bashisms without being obnoxious and annoying like that abomination that is dash. It's even smaller: seife@strolchi:/local/seife/src/git-repos/busybox> ls -lh _install/bin/ total 100K lrwxrwxrwx 1 seife users 7 Feb 8 19:02 ash -> busybox -rwxr-xr-x 1 seife users 100K Feb 8 19:02 busybox lrwxrwxrwx 1 seife users 7 Feb 8 19:02 sh -> busybox seife@strolchi:/local/seife/src/git-repos/busybox> ls -lh /usr/bin/dash -rwxr-xr-x 1 root root 124K Jan 30 02:39 /usr/bin/dash IMVHO dash serves only one purpose - as a bad example ;-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Friday 2019-02-08 18:10, Richard Brown wrote:

On Fri, 8 Feb 2019 at 13:23, Martin Wilck wrote:

...

I'd like to carry our OBS dicussion to a wider audience (https://build.opensuse.org/request/show/672510).

The question is whether we can assume that "/bin/sh" links to bash,

[...] But I think /bin/sh should be the bare minimum, smallest, leanest, most generic shell. I'm not sure it needs to be bash, I'm open to the idea of it changing, and I'm even willing to help with the insane amount of fallout that could be caused if we do decide to change it ;)

Thoughts?

Perhaps you can help your colleague markkp on SR 676611, he seems to have a severe difficulty in understanding the submission, particularly the sh part. See, I am just trying to reduce the aforementioned fallout before it happens. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Mischa Salle wrote:

I might have missed part of this (very long) discussion, but in my opinion - using /bin/sh means you SHOULD only use constructions which are POSIX compliant. - when you want to use bashisms, use /bin/bash. The same applies to other shells.

This is finally the right conclusion even thouh POSIX does not make any statements on what /bin/sh is. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Monday 2019-02-18 18:46, Martin Wilck wrote:

We need to define without ambiguity under what conditions a _shell script_ complies with the feature set of "/bin/sh". That means that we need to have a _test procedure_ (#). We could say that "code is compliant iff it runs without errors under every POSIX-compliant shell (*)". But that's hard to test

With a single prjconf line, we can make Factory (in a cloned project) build everything with an alternate sh. Haven't yet come to a selection whether or not I should do the initial build on a private instance, or start outright on build.opensuse.org. For essential runtime testing, there is OpenQA. Perhaps it's possible to add a job that derives from an existing run and has an extra action that switches sh via update-alternatives after installation, and then runs the rest of the testsuite. I don't quite know if OpenQA is capable of that yet. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 19/02/2019 04:16, Martin Wilck wrote:

On Mon, 2019-02-18 at 13:36 +0100, Joerg Schilling wrote:

...

Mischa Salle wrote:

...

I might have missed part of this (very long) discussion, but in my opinion - using /bin/sh means you SHOULD only use constructions which are POSIX compliant. - when you want to use bashisms, use /bin/bash. The same applies to other shells.

This is finally the right conclusion even thouh POSIX does not make any statements on what /bin/sh is.

I don't think that this is the right conclusion. Here's why.

I've been told in this discussion that the POSIX shell standard for "allows extensions". IOW, it only defines a minimal set of features, and implementations are free to add more features on top and still "comply" with the standard. Therefore the fact that bash, invoked as "/bin/sh", supports the non-POSIX '[[ ]]' syntax doesn't make it non- compliant.

We need to define without ambiguity under what conditions a _shell script_ complies with the feature set of "/bin/sh". That means that we need to have a _test procedure_ (#). We could say that "code is compliant iff it runs without errors under every POSIX-compliant shell (*)". But that's hard to test, as we'd have to test with every POSIX- compliant shell out there, and we may not even know them all. We also don't want to test by human beings parsing the code (even if that's basically what Jan is doing today). Rather, we need to settle on an actual shell implementation, and define "compliant" by reference to this implementation: "shell code is compliant iff it runs without errors under this implementation" (*).

That implementation could be bash in POSIX mode, meaning that e.g. '[[ ]]' would be allowed. Alternatively, it could be a minimal shell like "pbosh".

So far the discussion in this thread did not come up with a prime candidate for being that reference shell. dash has been propoposed and then harshly dismissed. ash, likewise. I suppose that Jörg would propose "pbosh", and it seems to come closer to the ideal of a "strict" POSIX shell as anything else mentioned up to now. But I'd call it a bit exotic. It doesn't seem to have a lot of real-world users under openSUSE.

I still fail to see the benefit of choosing anything else than bash in POSIX mode for /bin/sh, unless we want to pursue the long-term goal to be able to run (a minimal) openSUSE completely without bash.

Regards, Martin

(#) that doesn't mean we actually have to test every shell code snippet, but we _could_ in case of disagreement. (*) ... and yields correct results for a reasonable set of possible inputs.

I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features. Down the road if we decide as a project that we want to officially support multiple shells as #!/bin/sh I'd like to see debians checkers intergrated into rpmlint by the people who want to see this feature. I also don't think we need to have and agree on one explicit reference shell when I'm working on xdg-utils I generally go with use a subset of functionality thats in both bash and dash, maybe in openSUSE's case someone can make a compelling argument that pbosh should be added to the list then thats a case for people who are actually working on making it possible and wanting to run pbosh as #!/bin/sh to make. Not something we should add because some people believe it is the closest thing to the posix standard. We have to remember that the vast majority of the scripts we are talking about here are openSUSE / SUSE specific install scripts that wont make sense to run outside an openSUSE system and don't really need to be portable to the unix's of a bygone era. If the scripts don't fit that category then really they should be fixed wherever there upstream is. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Tue, Feb 19, 2019 at 11:09 AM Martin Wilck wrote:

On Tue, 2019-02-19 at 13:24 +1030, Simon Lees wrote:

...

I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features.

So - encourage people to remove bashisms on the one hand, and on the other hand allow more bashisms to creep in? As a policy, that doesn't make much sense to me.

We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged.

I see no point in going the Debian route. Forcing people to write scriptlets purely in POSIX is just adding pain for no benefit. The _only_ reason Debian enforces it is because they have a case where their init is a shell script. And even then, it's easy enough to argue otherwise since Bash continues to evolve and improve. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Tue, Feb 19, 2019 at 5:13 PM Neal Gompa wrote:

...

We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged.

I see no point in going the Debian route. Forcing people to write scriptlets purely in POSIX is just adding pain for no benefit. The _only_ reason Debian enforces it is because they have a case where their init is a shell script. And even then, it's easy enough to argue otherwise since Bash continues to evolve and improve.

No one is suggesting you have to write POSIX-compliant scripts, except in the case when you use #!/bin/sh. If you want to use any-other-shell-specifics, then specify that shell instead. In any case I would certainly go for option b) which allows the freedom to symlink /bin/sh to a faster performing or more POSIX compliant shell. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 3/12/19 3:41 PM, Ludwig Nussel wrote:

...

The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.

I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.

Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 07:28, Martin Wilck wrote:

On Tue, 2019-03-12 at 15:51 +0100, Stephan Kulow wrote:

...

On 3/12/19 3:41 PM, Ludwig Nussel wrote:

...

...

The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.

I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.

Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it

No it is not. A public discussion which was by no means settled is being ended the hard way, "creating facts". I can't imagine that that's what Richard meant. This "solution" is thoroughly lacking the "human touch" that Richard mentioned in his talk. It's also highly doubtful whether everyone involved in the discussion "feels they were heard". I, personally, do not. My key point in the discussion was that we need to settle on a unique, well-defined, existing shell with which scripts can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.

What I least understand is that this massive change is being rushed to factory in less than 2 weeks, while other things keep lurking in home projects or devel projects for ages. For example, in the course of this discussion, I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).

That was my approach to the subject, take some small steps, evaluate benefits, risks, and challenges, and then flip the switch (or not). But the big axe seems to be preferred. Well then. I've learned a not-so- nice lesson about openSUSE today.

I probably sound like a sore looser. Perhaps I am, although I'm not strongly on one side of the "pro-and-contra bash" discussion. I understand the "openSUSE is what you make it" mantra. But I dislike the way it's bluntly being applied here.

I tend to agree, with Martin here, I don't think such a change should be made until we can guarantee things wont break, and we can't do that until there is a way for packagers like me who have never cared what is / isn't a bashism to know they are breaking something as part of the buildsystem. I would request that those who are making such SR's stop until there is checks for bashism's as part of the buildsystem, otherwise i'll raise it to the board, as quite clearly at the moment the objections / concerns raised in the discussions on this list are not being taken into account in the current submissions. While I am generally ok with people making this change if they would like to I think it needs to be done right, and the current submission is not doing it right. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 08:33, Knurpht-openSUSE wrote:

...

On 13/03/2019 07:28, Martin Wilck wrote:

...

On Tue, 2019-03-12 at 15:51 +0100, Stephan Kulow wrote:

...

On 3/12/19 3:41 PM, Ludwig Nussel wrote:

...

...

The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.

I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.

Which is perfect! https://media.ccc.de/v/1912-opensuse-is-what-you-make-it

No it is not. A public discussion which was by no means settled is being ended the hard way, "creating facts". I can't imagine that that's what Richard meant. This "solution" is thoroughly lacking the "human touch" that Richard mentioned in his talk. It's also highly doubtful whether everyone involved in the discussion "feels they were heard". I, personally, do not. My key point in the discussion was that we need to settle on a unique, well-defined, existing shell with which scripts can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.

What I least understand is that this massive change is being rushed to factory in less than 2 weeks, while other things keep lurking in home projects or devel projects for ages. For example, in the course of this discussion, I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).

That was my approach to the subject, take some small steps, evaluate benefits, risks, and challenges, and then flip the switch (or not). But the big axe seems to be preferred. Well then. I've learned a not-so- nice lesson about openSUSE today.

I probably sound like a sore looser. Perhaps I am, although I'm not strongly on one side of the "pro-and-contra bash" discussion. I understand the "openSUSE is what you make it" mantra. But I dislike the way it's bluntly being applied here.

I tend to agree, with Martin here, I don't think such a change should be made until we can guarantee things wont break, and we can't do that until there is a way for packagers like me who have never cared what is / isn't a bashism to know they are breaking something as part of the buildsystem.

I would request that those who are making such SR's stop until there is checks for bashism's as part of the buildsystem, otherwise i'll raise it to the board, as quite clearly at the moment the objections / concerns raised in the discussions on this list are not being taken into account in the current submissions.

While I am generally ok with people making this change if they would like to I think it needs to be done right, and the current submission is not doing it right. Though not affected myself, I tend to agree with Martin as well. What about an

Op dinsdag 12 maart 2019 22:58:07 CET schreef Simon Lees: option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?

My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new (bashism vs normal shell) in order to not break things. Personally I think asking other people to learn new skills to care about things for you is not the best approach and doesn't fit with the openSUSE "Those who do decide" as lots of maintainers can decide not to learn and then we end up with something that is broken, whereas if a bot tells me i've screwed up in the same way a compiler does then its minimal effort for me to go and fix it. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 09:53, Jan Engelhardt wrote:

On Tuesday 2019-03-12 23:46, Simon Lees wrote:

...

...

Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?

My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new

My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box.

Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month.

So your adding an update-alternatives option that allows people to break there systems? I don't think that's any better. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 10:19, Patrick Shanahan wrote:

* Simon Lees [03-12-19 19:31]:

...

On 13/03/2019 09:53, Jan Engelhardt wrote:

...

On Tuesday 2019-03-12 23:46, Simon Lees wrote:

...

...

Though not affected myself, I tend to agree with Martin as well. What about an option in the installer "Preferred shell" ? Make bash default, but offer ( and install, when changed ) other shells?

My current main problem with the proposal is that it expects all contributors / members of the review team to go and learn something new

My problem is that you all seem to get waaay ahead of yourself. We are nowhere near the point where one could even think of adding a "preferred shell" dialog box.

Nor does the review team need to learn anything, as it is not a policy at this time to be sh conformant and the what-ifs were already brought up last month.

So your adding an update-alternatives option that allows people to break there systems? I don't think that's any better.

it is not a requirement to utilize update-alternatives in order to break one's system. if one does so in that manner, he has chosen do do so and you will not stop him by taking away that option.

If openSUSE is going to offer this as a supported option ie using update-alternatives rather then manually changing the symlink then we need to make sure it works, its all good for me and you to know it might make a mess of our systems but a general user who is not reading this list might see that the option has been added and make the reasonable presumption that it is supported and works which could lead to them breaking there system which is unacceptable. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 17:10, Stephan Kulow wrote:

Am 13.03.19 um 01:36 schrieb Simon Lees:

...

...

If openSUSE is going to offer this as a supported option

What does it bring if openSUSE supports an option? People file bug reports against packages - and maintainers of those close them WONTFIX as "send a patch, I don't support this myself".

Personally I don't think people should have to file potentially high severity bugs when it is not hard to prevent developers introducing said bugs in the first place and when fixing a bug will only ever take 1 line ie executing the script in bash rather then /bin/sh it is really not that much to expect them to fix it (currently the review team expects far more in other areas)

So please don't make more out of it as it is - it's not as your openSUSE salary depends on happy openSUSE users. First of all to break your system with this,

Well its a good thing my salary depends on happy SLE users instead, but the reality is I care about the quality of openSUSE, and I will call out developers who compromise the quality of openSUSE by taking lazy shortcuts rather then doing things properly.

other shells need to provide alternatives as well - and it's up to these shells to make sure their description says DO NOT EXPECT THIS TO WORK IN ALL USE CASES.

Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 23:23, Stephan Kulow wrote:

On 3/13/19 1:44 PM, Simon Lees wrote:

...

Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.

It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.

My point is more that I don't find it an acceptable acceptance level for openSUSE particularly when the amount of extra effort to make it much more reliable isn't large, but I guess we can agree to disagree.

Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.

Sure you can experiment with this it would take about 2 minutes and not much knowledge to create a repo in someones home dir where dash provided #!/bin/sh for example and bash did not once done they could even document it in a blog post for others to mess with. To me thats where this level of proposal belongs. On the other hand if one was to write an rpmlint style check to catch bashism's in #!/bin/sh code and then fixed the resulting errors i'd be happy to have the update-alternatives in openSUSE because the chances of it breaking systems would be significantly lower. Given such scripts already exist in debian land I don't think this is too much work and would provide a much more proper solution. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stephan Kulow schrieb:

On 3/13/19 1:44 PM, Simon Lees wrote:

...

Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.

It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.

Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.

If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage. On package level we have better options to control what goes where and what gets installed by default, eg by means of Requires in patterns or NON_FTP_PACKAGES. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thursday 2019-03-14 08:13, Ludwig Nussel wrote:

If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage.

Which is exactly what I've said elsewhere; comment 903827 within submit request 680550. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On czw, mar 14, 2019 at 8:13 AM, Ludwig Nussel wrote:

Stephan Kulow schrieb:

...

On 3/13/19 1:44 PM, Simon Lees wrote:

...

Yep and this is the big fat lazy option. Given the quality our release managers expect of our Tier 1 desktops and even our T2 desktops I don't see how we can include code "That will break in some cases" in the same category, if your only willing to go to that level of excellence keep the code in your home repo. Do you think such code is an acceptable standard for packagehub? personally I don't, and if its not acceptable for packagehub it should not be acceptable for Leap or Tumbleweed.

It's up to the packagehub maintainers to define their acceptance level, not on me. But I don't like the attitude to take freedom away from openSUSE users to break their systems. Breaking and fixing is basically the core of the fun people take out of linux.

Would I recommend anyone to switch /bin/sh? Hell, no! I personally would even go as far and block every alternative in the distribution - but with bash blocking /bin/sh there is not even a way for experiments without breaking your system integrity.

If we wanted to make it possible to replace /bin/sh without actually encouraging doing so in the distro I think it would be easier to have /bin/sh in a separate subpackage. On package level we have better options to control what goes where and what gets installed by default, eg by means of Requires in patterns or NON_FTP_PACKAGES.

I feel like it's time to draw a parallel between this and xdm/dm situation, where everybody has xdm installed for no particular reason :P LCP [Stasiek] https://lcp.world -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 12.03.19 um 21:58 schrieb Martin Wilck:

I'd been trying to help XRevan86 to get a working version of "checkbashisms" into factory, but so far it hasn't even received a devel project review ( https://build.opensuse.org/request/show/662123).

Please bring this up a little earlier - we have `factory-maintainers` group to help with devel project maintainers not being active anymore. Happens in all communities that contributors drop out - and we have measures not to make this a big problem. Greetings, Stephan -- Lighten up, just enjoy life, smile more, laugh more, and don't get so worked up about things. Kenneth Branagh -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, Mar 13, 2019 at 6:21 AM Joerg Schilling wrote:

Martin Wilck wrote:

...

can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.

I doubt that.

Currently, you have some obscure "wisdom" that there is something called "bash" where nobody is able to tell what features that covers since there is more than one single release of bash.

If you however write POSIX compliant scripts, you have the grant that in case there is a problem, you can file a bug against the installed shell.

For me, this looks like a big win.

This is not a win. This is a straight up fail. If you do this, you are allowing it to be possible to have a broken openSUSE bootstrap. You're allowing for people to choose an unsupported configuration in the very core of the distribution. That is completely nuts. Now admittedly, I don't like how much openSUSE uses alternatives for everything already, but at least we're not using it in ways where you'd potentially break your computer with a choice. This is literally making it so it's possible to have a failed openSUSE install. And even if the scriptlet was in Lua, there's other transactional ordering issues that would cause problems. It's much saner to just have bash as the supported /bin/sh and declare it as such. If you want to use a different shell for a scriptlet, specify it. This is insane. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 13 Mar 2019 11:20:28 +0100 Joerg Schilling wrote:

Martin Wilck wrote:

...

can be tested for compliance. After this change, we have the opposite. /bin/sh is now a black box that behaves "posix compliant" in some weakly defined way. Scripts may work, or they may not - no reliable way to find out.

I doubt that.

Currently, you have some obscure "wisdom" that there is something called "bash" where nobody is able to tell what features that covers since there is more than one single release of bash.

If you however write POSIX compliant scripts, you have the grant that in case there is a problem, you can file a bug against the installed shell.

Just as POSIX is a description of disparate implementations with various issues and the standard is updated over time as well. Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently. So from compatibility and reliability standpoint bash is clear win if you are in control of the shell used. And that is our case so long as we do not implement alternatives for /bin/sh. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow wrote:

On 3/13/19 1:56 PM, Michal Suchánek wrote:

...

Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.

And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.

It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build. Anyway, if we provide alternatives for /bin/sh you can assume any shell that claims to be vaguely bourne-compatible can end up as /bin/sh. All these shells have various bugs. Writing shell code that is bug-compatible with all shells the distribution ships is non-trivial. Some bugs are due to parser issues that are complex to fix and will not be resolved overnight. Ergo if we do provide alternatives for /bin/sh we cannot expect anything from /bin/sh anymore. Then you need to specify the interpreter for every shell script. Then there is no point in even having a /bin/sh. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 13 Mar 2019 14:30:36 +0100 Stephan Kulow wrote:

On 3/13/19 2:19 PM, Michal Suchánek wrote:

...

On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow wrote:

...

On 3/13/19 1:56 PM, Michal Suchánek wrote:

...

Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.

And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.

It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.Again, if you find yourself in need to write bash version specific %build scripts, write them in a real language - or at least put them in an extra script.

I am fine with it being any bash version form SLE11 to TW. As in if it builds everywhere it's ok. I am not fine if we switch it to 'any shell for which somebody installs an alternative'.

...

Anyway, if we provide alternatives for /bin/sh you can assume any shell

Yeah, that's right. *If* - *if* a shell in openSUSE wants to provide /bin/sh, it needs to eat all our rpm snippets that survived checkbashisms. Just because bash provides it, doesn't mean there are alternatives worth fighting over theoretic problems.

That's the problem. If people provide a broken /bin/sh alternative or one of the /bin/sh alternatives eventually regresses people will come complaining to me that my package does not build. If people provide two different /bin/sh alternatives with very different behavior in some corner cases I will need to rewrite my packaging scripts when I hit the corner case. I have been there writing scripts for Debian which provides several /bin/sh alternatives each with its unique set of bugs. I want no more of that.

And %build sections should potentially even be executed by /bin/bash --posix right away. But again, if your %build section is bash version specific, it's a lost case IMO.

You need not even use bash-specific syntax at all. You might just use some variable scoping that happens to work in bash and happens to not work in some other shell. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 13 Mar 2019 14:54:03 +0100 Marcus Hüwe wrote:

On 2019-03-13 14:19:23 +0100, Michal Suchánek wrote:

...

On Wed, 13 Mar 2019 13:59:08 +0100 Stephan Kulow wrote:

...

On 3/13/19 1:56 PM, Michal Suchánek wrote:

...

Bash has documentation. If implementation does not match documentation you can file a bug as much as you can against a POSIX-compliant shell. With bash being much more active project than most POSIX-compliant shells the bug is even much more likely to get resolved. And unlike POSIX-compliant shells bash has BASH_VERSION which tells you *exactly* what you are getting in case you need to support multiple versions that behave differently.

And if you're in need of writing shell scripts that are specific to bash versions, you're free to require /bin/bash as interpreter.

It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.

You can specify it via the %__spec_build_cmd macro. For instance, you could add a

%define __spec_build_cmd /bin/bash -e

Yes, that's helpful. If we get these alternatives I will add it to the compatibility cruft block together with the %license handling. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 2019-03-14 at 08:41 +0100, Dominique Leuenberger / DimStar wrote:

Just saying - Having a way out is good, opposing it for the sake of opposing it is unfriendly in a community.

The same can be said about pushing through a request that lots of people are opposing, just "because we can". Regards, Martin -- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

"Dr. Werner Fink" wrote:

I vote for configure the system Bourne SHell on the command line. As an example I provide the new package posh (Policy-compliant Ordinary SHell) in the project shells. With this the System Administrator has to do

If you start with this decision, you will fail with a high probability because posh is known to be full of bugs. There are better alternatives for an introducing test. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

"Dr. Werner Fink" wrote:

Which software is free of bugs ... OK, the final TeX/Web from Donald E. Knuth. The only known I see is the the well defined bugs of the posh:

Legitimate bugs are inconsistencies between manpage and behavior, and inconsistencies between behavior and Debian policy (currently SUSv3 compliance with the following exceptions: echo -n, binary -a and -o to test, local scoping).

which is OK IMHO :)

Do you believe, software generally documents all it's bugs in the man page? BTW: Even if you ignore the bugs related to e.g. "command" and shell arithmetik, posh is approx. 40% slower than pbosh or ksh93 and still 22% slower than bash if you use it to run "configure" fom gtar. An important fact here is that posh does not have a builtin "printf". Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 14 Mar 2019 08:41:24 +0100 Dominique Leuenberger / DimStar wrote:

On Wed, 2019-03-13 at 18:43 +0100, Michal Suchánek wrote:

...

You can specify it via the %__spec_build_cmd macro. For instance, you could

...

add a

%define __spec_build_cmd /bin/bash -e

Yes, that's helpful.

If we get these alternatives I will add it to the compatibility cruft block together with the %license handling.

And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.

The vast majoriti of packages uses "configure, make, makeinstall", "%cmake, %cmake_build,, %cmake_install"…

The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.

Just saying - Having a way out is good, opposing it for the sake of opposing it is unfriendly in a community.

Cheers, Dominique

PS: the official way of building packages for openSUSE and SLE will stay with osc build and OBS - And of course, should anybody ever have the idea of switching the defaulit shell in this setup, you can be sure that this won't happen without a staging pass.

If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore. And yes, I do not care for most packages that don't contain any convoluted code. Once you add a cycle you may run into variable scoping differences between shells, though. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 14 Mar 2019 12:25:03 +0100 (CET) Jan Engelhardt wrote:

On Thursday 2019-03-14 12:21, Michal Suchánek wrote:

...

If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore.

See the build log?

It's always /bin/sh. It does not show what it is linked to. Thanks MIchal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Mar 21, 2019 at 01:44:08AM +0100, Jan Engelhardt wrote:

On Thursday 2019-03-14 12:29, Michal Suchánek wrote:

...

On Thu, 14 Mar 2019 12:25:03 +0100 (CET) Jan Engelhardt wrote:

...

On Thursday 2019-03-14 12:21, Michal Suchánek wrote:

...

If the alternatives are implemented the shell is switchable *everywhere* including the OBS build environment. The dependencies may pull in a package that switches the shell. As I am not the reviewer of shell packages I cannot tell what's my package built with anymore.

See the build log?

It's always /bin/sh. It does not show what it is linked to.

If and when sh is under u-a control, u-a should be emitting something like

Setting up ... in auto mode

(or similar) when u-a is run as part of %post.

Except shell is special and u-a is not used. Thanks Michal

Moin, Am Mittwoch, 16. März 2022, 14:44:15 CET schrieb Dominique Leuenberger / DimStar:

On Wed, 2022-03-16 at 14:38 +0100, Michal Suchánek wrote:

...

...

If and when sh is under u-a control, u-a should be emitting something like

Setting up ... in auto mode

(or similar) when u-a is run as part of %post.

Except shell is special and u-a is not used.

minor correction: … not used *anymore* in tumbleweed (SLE15SP4 and Leap 15.4 will have the u-a based approach)

That's going to be reverted, 15SP4/15.4 are on track to use bash-sh like TW. Cheers, Fabian

this was actually only changed back in * Fri Oct 22 2021 by Stefan Schubert - Using package bash-sh instead of the update-alternative mechanism.

But we still have multiple xxx-sh packages that are mutually exclusive and that install the /bin/sh -> /bin/xxx symlink as needed (currently we have bash-sh, dash-sh, and busybox-sh)

Cheers, Dominique

Moin, Am Mittwoch, 16. März 2022, 15:19:12 CET schrieb Michal Suchánek:

Hello,

On Wed, Mar 16, 2022 at 03:05:26PM +0100, Fabian Vogt wrote:

...

Moin,

Am Mittwoch, 16. März 2022, 14:44:15 CET schrieb Dominique Leuenberger / DimStar:

...

On Wed, 2022-03-16 at 14:38 +0100, Michal Suchánek wrote:

...

...

If and when sh is under u-a control, u-a should be emitting something like

Setting up ... in auto mode

(or similar) when u-a is run as part of %post.

Except shell is special and u-a is not used.

minor correction: … not used *anymore* in tumbleweed (SLE15SP4 and Leap 15.4 will have the u-a based approach)

That's going to be reverted, 15SP4/15.4 are on track to use bash-sh like TW.

And are there u-a based alternatives before that or was it always hardcodedto bash?

FWICT SLE 15 SP3 and below had bash providing /bin/sh. Cheers, Fabian

Thanks

Michal

On 3/14/19 12:46 PM, Stefan Seyfried wrote:

Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:

...

The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.

Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P

Yast team has experience in converting strange languages to each other - they can cook something! :) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 14 Mar 2019 12:46:06 +0100 Stefan Seyfried wrote:

Am 14.03.19 um 08:41 schrieb Dominique Leuenberger / DimStar:

...

The review team will be ok if you switch the interpreter from to explicit bash IFF there is need for. But please, don't just add this prophilactic to any spec file just because you oppose something.

Would it be ok to just switch all build sections to use lua? Probably much faster than the shells anyway :-P

How much time is spent executing the spec files, anyway? Who would write those lua scripts? There is quite a disproportion between the number of people who understand shell and the number of people who understand lua. Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 14.03.19 um 13:12 schrieb Michal Suchánek:

How much time is spent executing the spec files, anyway?

Who would write those lua scripts?

I, for "my" packages. I'm not proposing to do this for all packages. I'm just interested if there would be objections if I converted all the packages that I maintain to use lua instead of shell.

There is quite a disproportion between the number of people who understand shell and the number of people who understand lua.

Oh! A second advantage. No more fiddling around in "my" specfiles by people running "spec-cleaner" or other crap. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Mar 14, 2019 at 08:41:24AM +0100, Dominique Leuenberger / DimStar wrote:

On Wed, 2019-03-13 at 18:43 +0100, Michal Suchánek wrote:

...

You can specify it via the %__spec_build_cmd macro. For instance, you could

...

add a

%define __spec_build_cmd /bin/bash -e

Yes, that's helpful.

If we get these alternatives I will add it to the compatibility cruft block together with the %license handling.

And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.

So now I have recieved this patch for kernel: commit aaf726d5cc067139fd2ade9d4fc1ab1c3e4a268d (origin/users/dmueller/packaging/for-next) Author: Dirk Müller Date: Wed Mar 9 21:45:15 2022 +0100 - rpm/*.spec.in.in: Avoid bashism in subshells (bsc#1195391) Maybe it's time for %define __spec_build_cmd /bin/bash -e Maybe that is not sufficient because there is also shell used for %(). Thanks Michal

On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:

Hi Michal,

Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek :

...

...

...

You can specify it via the %__spec_build_cmd macro. For instance, you could

...

add a %define __spec_build_cmd /bin/bash -e

This is actually pullrequested already for a while, see here:

https://github.com/openSUSE/rpm-config-SUSE/pull/53

you don't need to worry about not using bash as the build shell. there are two cases where it could change:

* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()

...

...

And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.

There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.

...

So now I have recieved this patch for kernel:

Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.

And rather than making the code more complex I would opt for BuildRequires: bash-sh that way there will be no surprises even for %(). While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable. Thanks Michal

Hello, On Thu, Mar 17, 2022 at 10:06:09PM +0100, Dirk Müller wrote:

...

...

Yes, because it fixes the subshell usage of being a bit more compatible on a wider range. And rather than making the code more complex I would opt for

Is the code really more complex by this change?

-%(chmod +x %_sourcedir/{@SCRIPTS@}) +%(cd %_sourcedir && chmod +x @SCRIPTS@)

slightly

That's the bulk of the changes. I think the simpler chmod statement makes that one actually a tad easier to read, as you don't have to carefully parse brackets inside parentheses to understand what is going on.

That's completely subjective

The other one is using unquoted " within " which is even recommended against in bash, where a single quote would work with any shell.

I am not against improving readability of shell code in general, and this may fall in that category

I can also carry this as a local patch, but it is so minor that I don't think it adds any significant complexity. we're talking about 4 additional characters that, depending on how you see it, remove a {} construct that might be less obvious to understand also.

...

BuildRequires: bash-sh

that way there will be no surprises even for %().

that would make the kernel unresolvable and I'd have to carry a local patch to revert that so it would only make my work even harder than it already is. Also this would only work within the build service and would not make anything different for anyone using any other method of building the rpms.

So long as you build rpms the dependencies should be fulfilled so it will work for any method that builds rpms. If you don't build rpms you supply your own build scripts and you are free to write them in any language you choose.

...

While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system.

I agree with that. In order to test that it is installable on a minimal system in the buildservice, I need to have /bin/sh that way,

Buildservice is a build environment, not a test environment. By testing in buildservice you are conflating build time and runtime. Please use the correct tool for the job. It's not like we don't have any.

because packages are installed at the end of the build and have their post install scripts executed.

I'm near the bottom of this effort, the kernel is one of the last packages that still fails. It works for almost everything (I have about 10 failures including the kernel packages left of the ~ 3000 packages in the ring).

I suppose those 10 are those that actually use non-trivial scripting during the build.

I am not patching out complex and useful bash features here. These are hair sized changes that do not harm the understandability of the code. I'm not asking for a lot here, and while I do have a workaround of carrying a local patch and keeping the kernel that way, that costs me significantly more continuous work than upstreaming the change. And the change is not making your hard maintenance work any harder. you don't have to keep this working at all, and the lines I modified haven'T been touched in the last 11 years in git so they're certainly also not a big maintenance complexity for anyone else.

And when one of the alternative shells subtly breaks some of the code that is not explicitly executed in bash because there is no way to specify the interpreter for it will you also request that the kernel carries compatibility cruft for such breakage? Where do you draw the line? I don't support building with anything but bash, to reduce maintenance effort. Now that the distribution grew /bin/sh alternatives I will request that for kernel build /bin/sh is bash, always. Thanks Michal

Hello On Fri, Mar 18, 2022 at 10:14:30PM +1030, Simon Lees wrote:

On 3/18/22 06:51, Michal Suchánek wrote:

...

On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:

...

Hi Michal,

Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek :

...

...

...

You can specify it via the %__spec_build_cmd macro. For instance, you could > add a > %define __spec_build_cmd /bin/bash -e

This is actually pullrequested already for a while, see here:

https://github.com/openSUSE/rpm-config-SUSE/pull/53

you don't need to worry about not using bash as the build shell. there are two cases where it could change:

* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()

...

...

And I'm sure the review team will decline them if there is no proper reason to do this - if you don't have at least *some* complexity in the build section.

There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.

...

So now I have recieved this patch for kernel:

Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.

And rather than making the code more complex I would opt for

BuildRequires: bash-sh

that way there will be no surprises even for %().

While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.

As someone who cares about the build time of packages like the kernel I would be interested in looking at the differences in build time between using bash and dash as /bin/sh my suspicion is that the kernel build process still launches /bin/sh enough times that using a lighter alternative might make a difference. So if its not adding massive complexity i'd also appreciate keeping support for multiple /bin/sh here atleast until we can do better analysis on it.

Until better analysis is done I see this as a case of premature optimization. My suspicion is that the runtime of the scripts that are not run by bash explicitly is irrelevant. Also there is no intention to change the default shell in the distributions so any savings by lighter shell will be obtained only in test projects. Thanks Michal

On Sat, Mar 19, 2022 at 03:03:47PM +1030, Simon Lees wrote:

On 3/19/22 03:15, Michal Suchánek wrote:

...

Hello

On Fri, Mar 18, 2022 at 10:14:30PM +1030, Simon Lees wrote:

...

On 3/18/22 06:51, Michal Suchánek wrote:

...

On Thu, Mar 17, 2022 at 07:29:51PM +0100, Dirk Müller wrote:

...

Hi Michal,

Am Mi., 16. März 2022 um 14:50 Uhr schrieb Michal Suchánek :

...

>> You can specify it via the %__spec_build_cmd macro. For instance, you could >>> add a >>> %define __spec_build_cmd /bin/bash -e

This is actually pullrequested already for a while, see here:

https://github.com/openSUSE/rpm-config-SUSE/pull/53

you don't need to worry about not using bash as the build shell. there are two cases where it could change:

* when somebody forcefully changes /bin/sh to find issues where we depend on bash * when shell invocation are used, like %()

...

> And I'm sure the review team will decline them if there is no proper > reason to do this - if you don't have at least *some* complexity in the > build section.

There is no point in changing this for build, that would be entirely madness, there are thousands of packages that rely on bashism.

...

So now I have recieved this patch for kernel:

Yes, because it fixes the subshell usage of being a bit more compatible on a wider range.

And rather than making the code more complex I would opt for

BuildRequires: bash-sh

that way there will be no surprises even for %().

While the kernel should be installable on a minimal system I don't see any reason why it should be buildable on a minimal system. Surely the fact that kernel requires gcc (specific version, not clang), bison, openssl or dwarves to build does not offend anybody. In the same way I see no problem with requiring bash for build when it makes the build scripts simpler and more reliable.

As someone who cares about the build time of packages like the kernel I would be interested in looking at the differences in build time between using bash and dash as /bin/sh my suspicion is that the kernel build process still launches /bin/sh enough times that using a lighter alternative might make a difference. So if its not adding massive complexity i'd also appreciate keeping support for multiple /bin/sh here atleast until we can do better analysis on it.

Until better analysis is done I see this as a case of premature optimization. My suspicion is that the runtime of the scripts that are not run by bash explicitly is irrelevant.

Well Dirk's changes will allow us to properly analyse this.

Will it? The kernel is built on different builder each time with build times varying wildly. Running a build with different shell once or twice gives absolutely no insight into the impact of using different shells. Thanks Michal

On 3/20/22 19:57, Jan Engelhardt wrote:

On Saturday 2022-03-19 13:39, Michal Suchánek wrote:

...

...

Well Dirk's changes will allow us to properly analyse this.

Will it?

The kernel is built on different builder each time with build times varying wildly. Running a build with different shell once or twice gives absolutely no insight into the impact of using different shells.

Add a _constraint that limits it to a particular worker.

Yeah or what we generally do is build multiple times for each configuration and then we can work out what the difference is on both the slower and faster workers. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

On 2019-03-14 12:17:25 +0100, Jan Engelhardt wrote:

On Wednesday 2019-03-13 14:54, Marcus Hüwe wrote:

...

...

It is not always clear how to do that. I thought there is a way to switch rpm spec scriptlets to different shell but did not find how to do that for %build.

You can specify it via the %__spec_build_cmd macro. For instance, you could add a

%define __spec_build_cmd /bin/bash -e

line to your spec file.

That looks really insance. Why not just do what specfiles already do?

%define _buildshell /bin/bash

That would also influence %prep, %install etc. I just realized that __spec_build_shell might be the better choice if one wants to change build shell only. However, in all cases (%_buildshell, %__spec_build_cmd, %__spec_build_shell), one should also consider the __spec_build_template and related macros... Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Wilck wrote:

I don't disagree. This is one argument for using a different shell, which is in "feature freeze" mode. OTOH, the changes between bash 4.x and 5.0 are rather insignificant in terms of shell script syntax, AFAICT.

I remember that we recently discoverd a significant difference between bash4 and bash4 while discussing shell behavior in the POSIX standard teleconference. Sorry, I no longer remember what exactly this was related to.

I for one wouldn't want to rely on my brain's built-in Posix shell code parser. I'd much prefer to have a computer do it for me.

I do it this way: I primarily believe that the parser in bosh is correct, since it eveolved since more than 42 years. Whenever I see a problem, I check the behavior of ksh88, ksh93 and bosh and then decide using my brain what is correct. The trick behind that method is that all three shells are direct descendents from the original Bourne Shell (which is a good starter) and their maintainers tried to do their best to fix problems in the behavior of the original Bourne Shell. Writing a correct shell is definitely not trivial and other shells that have been written from scratch usually have more or less deviations. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Tue, Mar 12, 2019 at 10:41 AM Ludwig Nussel wrote:

Martin Wilck schrieb:

...

On Tue, 2019-02-19 at 13:24 +1030, Simon Lees wrote:

...

I think that as a starting point we might not even need to go this far. A simple policy of "Contributors should accept submissions that remove bashism's from scripts with a #!/bin/sh shebang or replace it with #!/bin/bash" is reasonable. That way we are not blocking people who want to work on this from working on it. Whether the shebang should be replaced or the code is modified to be more compliant should be up to the maintainer as they are the person who has to maintain the code long term. At the same time submissions to factory shouldn't be blocked because scripts in #!/bin/sh are using shell specific features.

So - encourage people to remove bashisms on the one hand, and on the other hand allow more bashisms to creep in? As a policy, that doesn't make much sense to me.

We should either say a) "YES, /bin/sh is always bash on openSUSE" (like Fedora), or b) "/bin/sh could be any POSIX compliant shell" (like Debian). In case a), removing bashisms would be pointless and wrong and should therefore be discouraged. If we go for b), we must avoid bashisms in new submissions. Even the syntax "%post -p /bin/bash" should be considered transitional in that case.

IMO it has to be either a) or b) eventually. I see no benefit in looking for a middle ground (other than for transition).

...

Down the road if we decide as a project that we want to officially support multiple shells as #!/bin/sh I'd like to see debians checkers intergrated into rpmlint by the people who want to see this feature.

The longer this thread evolves, the more I wonder how a decision could be reached. But I do think we need one. Otherwise we'll waste energy in pointless races where some people remove bashisms and others introduce new ones. We all have better things to do.

I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.

That creates a dependency loop, though. Managing the core shell through alternatives is nuts because it'll break things for stateless things, require a scriptlet engine to run the program, and the scriptlet engine isn't configured yet because alternatives isn't set up... -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Tuesday 2019-03-12 17:23, Neal Gompa wrote:

...

I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.

That creates a dependency loop, though.

bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.

Managing the core shell through alternatives is nuts because it'll break things for stateless things, require a scriptlet engine to run the program

A trivial look shows that update-alternatives is an ELF binary. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Mär 13 2019, Stefan Seyfried wrote:

but %{pre,post,preun,postun,...} are executed with the help of a shell

Not necessarily. They can also be implemented in embedded lua. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Mar 14, 2019 at 12:01:16AM +0100, Jan Engelhardt wrote:

On Wednesday 2019-03-13 09:50, Andreas Schwab wrote:

...

On Mär 13 2019, Stefan Seyfried wrote:

...

but %{pre,post,preun,postun,...} are executed with the help of a shell

Not necessarily. They can also be implemented in embedded lua.

And we could also just (maybe) write

%pre -p /usr/sbin/update-alternatives --install ...

Exciting how rpm works.

It should be mentioned that with e.g. %post -p <lua> the os.execute() and io.popen() lua API do use /bin/sh just like (g)libc system(3), that is ther is no os.execve() :P Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr

On Thu, Mar 14, 2019 at 10:44:01AM +0100, Stephan Kulow wrote:

Am 14.03.19 um 07:17 schrieb Dr. Werner Fink:

...

It should be mentioned that with e.g.

%post -p <lua>

the os.execute() and io.popen() lua API do use /bin/sh just like (g)libc system(3), that is ther is no os.execve() :P

posix.exec is added by rpm

Woh ... luxus pur :D Werner -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr

On Mär 14 2019, Jan Engelhardt wrote:

And we could also just (maybe) write

%pre -p /usr/sbin/update-alternatives --install ...

Exciting how rpm works.

I can't pass arguments to the interpreter here. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13/03/2019 20:55, Joerg Schilling wrote:

Jan Engelhardt wrote:

...

On Tuesday 2019-03-12 17:23, Neal Gompa wrote:

...

...

I can't see a consensus reached in this discussion. Yet, a request to put /bin/sh under control of update-alternatives is on the way to Factory. So just creating facts.

That creates a dependency loop, though.

bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.

I am not sure what you mean by that, but bash can be compiled as "POSIX compliant". This is e.g. done on Mac OS (where bash is installed as /bin/sh and where no other POSIX shell is provided by the OS) and on Solaris, where bash is just installed as /usr/bin/bash (with a symlink /usr -> /usr/bin) and where bash is just one of many shells.

If you do that, you break all noncompliant scripts that typically make non-POSIX assumptions on the behavior of the built in "echo" command.

Given that posix has extensions and all the shells have some wriggle room to do things slightly differently, having something like the subset of features that work on shell X, Y and Z is far more useful and actually testable where if you just say "posix" people could choose to interpret that as bash --posix which won't necessarily work on dash so the point is we need to create an openSUSE definition of posix that is clear and everyone understands such a definition could be "the subset of commands / syntax that works on both bash and dash", if there are other shells that people care about that break under that definition its simple to add them. Then everything is clear and easy bots can be written etc... -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 14/03/2019 00:52, Martin Wilck wrote:

On Wed, 2019-03-13 at 22:58 +1030, Simon Lees wrote:

...

the point is we need to create an openSUSE definition of posix that is clear and everyone understands such a definition could be "the subset of commands / syntax that works on both bash and dash",

I'd rather settle down on _one_ shell to test with.

Martin

That indeed would be nice, but I don't think its really possible without writing another reference shell (or forking an existing one to remove features until we get close enough). However there exists checkers that catch most if not all of the common errors and having them setup in the build process to do the majority of testing automatically (so in general packagers need to do nothing) seems like a better approach, then in case someone really wants to argue semantics we can point them to does the script run in bash and dash, if yes its acceptable if not its not for example. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wednesday 2019-03-13 11:25, Joerg Schilling wrote:

...

bash already pulled in update-alternatives (at least for build time) already in some way, so using it seemed more or less free.

I am not sure what you mean by that

The build log for bash.rpm showed that "update-alternatives" was already installed as a build-time dependency to create openSUSE's bash. So the particular situation that a build cycle already existed was not made worse. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Thorsten Kukuk wrote:

The problem is the installation dependency cycle: - bash requires update-alternatve - update-alternative requires bash

If you install bash first, the update-alternative call in the %post section will fail and there is no /bin/sh.

This is not a problem as long as you install bash as /bin/bash and include bash in a packet that does not depend on things that bas does not need to depend on.

What I'm still missing is: what is the advantage to be able to replace /bin/sh with something different than bash? bash will be installed anyways, there is no way to install an openSUSE without bash, there are too many core packages requiring /bin/bash.

Other shells are e.g. faster and this could speed up the boostrap process. bosh is typically 20% faster than bash and still 4% faster than dash even though bosh implements the POSIX required support for multi byte characters. In general, it is a nice idea to prevent people from introducing constructs into a "#!/bin/sh script" that are not granted to be in a shell since in general it is no longer a correct assumption to have bash in /bin/sh on Linux. In other words: performance and portability is enhanced by such a decision. Jörg -- EMail:joerg@schily.net (home) Jörg Schilling D-13353 Berlin joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/' -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org