RFC: changing default build environment to use only glibc-locale-base

newer
New package: perl-File-Rename —...

older
New Tumbleweed snapshot 20220130...

Dirk Müller

28 Jan 2022 28 Jan '22

14:43

Hi all, Some time ago I noticed that Fedora is building all their packages with very minimal locales installed quite successfully. In fedora, if there is the occasional packaging testsuite that requires a specific locale being installed, Fedora just uses "BuildRequires: glibc-langpack-$lang". openSUSE currently doesn't have this granularity. we have glibc-locale-base, which covers C and en_US locale and we have glibc-locale which covers everything else. The vast majority of packages are building and testing fine in the default C.utf8/en_US.utf8 call. By switching to glibc-locale-base, we're saving about 227 MB of data to pull and install on every build. That's like 2TB of data less written for a full distro build. if you're behind crappy internet like a german citizen would be, then you love the speedup. Any comments, concerns? Yes I'm aware a few packages will fail to build. I've done a full rebuild of :Ring:0-Bootstrap/1-Minimalx and had about 10 build failures, and submitted already fixes for all of them. I submitted this to rpm maintainer for review, and he's fine if the opensuse tumbleweed distro team accepts the challenge. I'm happy to help with the fallout, the simplest fix is an additional "Buildrequires: glibc-locale". In some other cases it just requires to change the locale that is exported to a locale that is provided by glibc-locale-base https://build.opensuse.org/request/show/949701 Comments, feedback, objections? Thanks, Dirk

Show replies by date

Guillaume Gardet

28 Jan 28 Jan

16:31

Hi Dirk,

...

-----Original Message----- From: Dirk Müller Sent: 28 January 2022 15:43 To: factory@lists.opensuse.org Subject: RFC: changing default build environment to use only glibc-locale-base

Hi all,

Some time ago I noticed that Fedora is building all their packages with very minimal locales installed quite successfully. In fedora, if there is the occasional packaging testsuite that requires a specific locale being installed, Fedora just uses "BuildRequires: glibc-langpack-$lang".

openSUSE currently doesn't have this granularity. we have glibc-locale-base, which covers C and en_US locale and we have glibc-locale which covers everything else.

The vast majority of packages are building and testing fine in the default C.utf8/en_US.utf8 call. By switching to glibc-locale-base, we're saving about 227 MB of data to pull and install on every build. That's like 2TB of data less written for a full distro build. if you're behind crappy internet like a german citizen would be, then you love the speedup.

Any comments, concerns? Yes I'm aware a few packages will fail to build. I've done a full rebuild of :Ring:0-Bootstrap/1-Minimalx and had about 10 build failures, and submitted already fixes for all of them.

I submitted this to rpm maintainer for review, and he's fine if the opensuse tumbleweed distro team accepts the challenge. I'm happy to help with the fallout, the simplest fix is an additional "Buildrequires: glibc-locale". In some other cases it just requires to change the locale that is exported to a locale that is provided by glibc-locale-base

https://build.opensuse.org/request/show/949701

Comments, feedback, objections?

That sounds good! It will likely help to speed-up builds. Thanks, Guillaume

...

Thanks, Dirk

Cristian Rodríguez

21:37

On Fri, Jan 28, 2022 at 11:43 AM Dirk Müller wrote: < if

...

you're behind crappy internet like a german citizen would be, then you love the speedup.

I am behind fast internet but osc seems to talk to snail mirrors so this change actually helps me too.

...

Any comments, concerns? Yes I'm aware a few packages will fail to build. I've done a full rebuild of :Ring:0-Bootstrap/1-Minimalx and had about 10 build failures, and submitted already fixes for all of them.

Huh.. shouldn't the locale be C on every build step ?

Neal Gompa

21:40

On Fri, Jan 28, 2022 at 4:37 PM Cristian Rodríguez wrote:

...

On Fri, Jan 28, 2022 at 11:43 AM Dirk Müller wrote: < if

...
you're behind crappy internet like a german citizen would be, then you love the speedup.

I am behind fast internet but osc seems to talk to snail mirrors so this change actually helps me too.

...
Any comments, concerns? Yes I'm aware a few packages will fail to build. I've done a full rebuild of :Ring:0-Bootstrap/1-Minimalx and had about 10 build failures, and submitted already fixes for all of them.

Huh.. shouldn't the locale be C on every build step ?

OBS forces a weird "POSIX" locale right now, rather than C.UTF-8. That causes a bunch of problems in itself, too. :( -- 真実はいつも一つ！/ Always, there's only one truth!

Jan Engelhardt

23:42

On Friday 2022-01-28 22:37, Cristian Rodríguez wrote:

...

On Fri, Jan 28, 2022 at 11:43 AM Dirk Müller wrote: < if

...
you're behind crappy internet like a german citizen would be, then you love the speedup.

I am behind fast internet but osc seems to talk to snail mirrors so this change actually helps me too.

It might be more of a concern for bs_worker-type workers than it is for an osc-type worker, since osc certainly keeps a cache in /var/tmp/osbuild-packagecache.

...

Huh.. shouldn't the locale be C on every build step ?

Should, yes. But you need only grep for LC_CT across factory and find... httpie.spec:export LC_CTYPE=en_US.UTF-8 libmodulemd.spec:export LC_CTYPE=C.utf8 log4net.spec:export LC_CTYPE=en_US.UTF-8 procps.spec:unset LC_CTYPE python-eliot.spec:export LC_CTYPE=en_US.UTF-8 python-Flask-Gravatar.spec:export LC_CTYPE=en_US@UTF-8 python-Flask-Migrate.spec:export LC_CTYPE=en_US.UTF-8 python-nose2.spec:export LC_CTYPE=C.UTF8 readline6.spec: unset LC_CTYPE readline.spec:unset LC_CTYPE ruby2.7.spec:export LC_CTYPE="en_US.UTF-8" ruby3.0.spec:export LC_CTYPE="en_US.UTF-8" texlive.spec: echo LC_CTYPE=en_US.UTF-8 texlive.spec: echo export LANG LC_CTYPE words.spec: american*) LC_CTYPE=en_US.UTF-8 ;; words.spec: british*) LC_CTYPE=en_GB.UTF-8 ;; words.spec: canadian*) LC_CTYPE=en_CA.UTF-8 ;; this can only be described by a medium not carryable by plaintext email, https://en.meming.world/wiki/Cat_Standing_in_the_Snow Yes, especially the "@UTF-8" one.

Neal Gompa

29 Jan 29 Jan

00:09

On Fri, Jan 28, 2022 at 6:43 PM Jan Engelhardt wrote:

...

On Friday 2022-01-28 22:37, Cristian Rodríguez wrote:

...
On Fri, Jan 28, 2022 at 11:43 AM Dirk Müller wrote: < if

...
you're behind crappy internet like a german citizen would be, then you love the speedup.

I am behind fast internet but osc seems to talk to snail mirrors so this change actually helps me too.

It might be more of a concern for bs_worker-type workers than it is for an osc-type worker, since osc certainly keeps a cache in /var/tmp/osbuild-packagecache.

...
Huh.. shouldn't the locale be C on every build step ?

Should, yes. But you need only grep for LC_CT across factory and find...

httpie.spec:export LC_CTYPE=en_US.UTF-8 libmodulemd.spec:export LC_CTYPE=C.utf8 log4net.spec:export LC_CTYPE=en_US.UTF-8 procps.spec:unset LC_CTYPE python-eliot.spec:export LC_CTYPE=en_US.UTF-8 python-Flask-Gravatar.spec:export LC_CTYPE=en_US@UTF-8 python-Flask-Migrate.spec:export LC_CTYPE=en_US.UTF-8 python-nose2.spec:export LC_CTYPE=C.UTF8 readline6.spec: unset LC_CTYPE readline.spec:unset LC_CTYPE ruby2.7.spec:export LC_CTYPE="en_US.UTF-8" ruby3.0.spec:export LC_CTYPE="en_US.UTF-8" texlive.spec: echo LC_CTYPE=en_US.UTF-8 texlive.spec: echo export LANG LC_CTYPE words.spec: american*) LC_CTYPE=en_US.UTF-8 ;; words.spec: british*) LC_CTYPE=en_GB.UTF-8 ;; words.spec: canadian*) LC_CTYPE=en_CA.UTF-8 ;;

this can only be described by a medium not carryable by plaintext email, https://en.meming.world/wiki/Cat_Standing_in_the_Snow Yes, especially the "@UTF-8" one.

It's because OBS does not set the environment to "C.UTF-8", but "POSIX" instead (which is an alias of the "C" locale, and an alias that not every application understands). That breaks things sometimes, so it needs to be explicitly set. -- 真実はいつも一つ！/ Always, there's only one truth!

Cristian Rodríguez

15:27

On Fri, Jan 28, 2022 at 9:09 PM Neal Gompa wrote:

...

It's because OBS does not set the environment to "C.UTF-8", but "POSIX" instead (which is an alias of the "C" locale, and an alias that not every application understands).

So, which applications are these ? I ask because I have a pyre to lit.. :-) understanding "C" or "POSIX" is mandatory... That breaks things sometimes, so

...

it needs to be explicitly set.

Looks like something that really needs fixing..

Neal Gompa

15:43

On Sat, Jan 29, 2022 at 10:27 AM Cristian Rodríguez wrote:

...

On Fri, Jan 28, 2022 at 9:09 PM Neal Gompa wrote:

...
It's because OBS does not set the environment to "C.UTF-8", but "POSIX" instead (which is an alias of the "C" locale, and an alias that not every application understands).

So, which applications are these ? I ask because I have a pyre to lit.. :-) understanding "C" or "POSIX" is mandatory...

That breaks things sometimes, so

...
it needs to be explicitly set.

Looks like something that really needs fixing..

No. The C/POSIX locales also disable Unicode support, which can break some applications. That's why Fedora had a patch to force Python to coerce C to C.UTF-8 before it was finally upstreamed in Python 3.8. The C locale is woefully bad and broken for modern applications, but because of POSIX, we can't change the existing C locale to be Unicode enabled, so we now have C.UTF-8. -- 真実はいつも一つ！/ Always, there's only one truth!

Dirk Müller

17:25

Hi all, Am Sa., 29. Jan. 2022 um 16:44 Uhr schrieb Neal Gompa :

...

No. The C/POSIX locales also disable Unicode support, which can break some applications. That's why Fedora had a patch to force Python to coerce C to C.UTF-8 before it was finally upstreamed in Python 3.8.

This is not relevant to the question whether glibc-locale-base should be used instead of glibc-locale. I'm not changing any locale choices at this time. Greetings, Dirk

Neal Gompa

17:41

On Sat, Jan 29, 2022 at 12:26 PM Dirk Müller wrote:

...

Hi all,

Am Sa., 29. Jan. 2022 um 16:44 Uhr schrieb Neal Gompa :

...
No. The C/POSIX locales also disable Unicode support, which can break some applications. That's why Fedora had a patch to force Python to coerce C to C.UTF-8 before it was finally upstreamed in Python 3.8.

This is not relevant to the question whether glibc-locale-base should be used instead of glibc-locale. I'm not changing any locale choices at this time.

Well, it is. In Fedora, we were able to move to glibc-minimal-langpack in our buildroot because we did work to coerce the default locale in some stacks to C.UTF-8, which eliminated the need for "regular" locales for most package builds. It started with Python and work extended from there. https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale -- 真実はいつも一つ！/ Always, there's only one truth!

Christian Boltz

18:25

Hello, Am Samstag, 29. Januar 2022, 00:42:52 CET schrieb Jan Engelhardt:

...

On Friday 2022-01-28 22:37, Cristian Rodríguez wrote:

...
Huh.. shouldn't the locale be C on every build step ?

Should, yes. But you need only grep for LC_CT across factory and find...

...

httpie.spec:export LC_CTYPE=en_US.UTF-8 libmodulemd.spec:export LC_CTYPE=C.utf8 log4net.spec:export LC_CTYPE=en_US.UTF-8 [...]

Most of your grep results are en_US.utf8 or C.utf8, and the good thing is that /usr/lib/locale/en_US.utf8 and /usr/lib/locale/C.utf8 are part of glibc-locale-base. So even with glibc-locale-base, these packages shouldn't have any build issues. The only exception from your list is

...

words.spec: american*) LC_CTYPE=en_US.UTF-8 ;; words.spec: british*) LC_CTYPE=en_GB.UTF-8 ;; words.spec: canadian*) LC_CTYPE=en_CA.UTF-8 ;;

and since there are words-british and words-canadian subpackages, I'm not too surprised that it uses en_GB and en_CA. (However, I'm surprised that this is needed in the spec.)

...

this can only be described by a medium not carryable by plaintext email, https://en.meming.world/wiki/Cat_Standing_in_the_Snow

I think my (random!) signature can easily beat this picture ;-)

...

Yes, especially the "@UTF-8" one.

Just curious - what's the difference between ".UTF-8" and "@UTF-8"? Or is "@UTF-8" just wrong and broken? (Needless to say: There's no "*@UTF-8" in /usr/lib/locale/ on my system, with the full glibc-locale installed.) Regards, Christian Boltz --

...

AAAAAAAAAHHHHHHHHHHHHHH *inDieTastaturBeiß* ^^^^^^^^^^^^^ Mahlzeit! [> Michael Born und Kara in opensuse-de]

(For those not speaking german: "inDieTastaturBeiß" means "biting the keyboard", and "Mahlzeit" translates to "enjoy your meal".)

Cristian Rodríguez

23:34

On Sat, Jan 29, 2022 at 3:25 PM Christian Boltz wrote:

...

Just curious - what's the difference between ".UTF-8" and "@UTF-8"? Or is "@UTF-8" just wrong and broken? (Needless to say: There's no "*@UTF-8" in /usr/lib/locale/ on my system, with the full glibc-locale installed.)

it is not a valid locale name... locale -a | grep @ for the locales where the @ symbol is valid.

Dominique Leuenberger / DimStar

1 Feb 1 Feb

10:16

Hi Dirk, On Fri, 2022-01-28 at 15:43 +0100, Dirk Müller wrote:

...

Hi all,

Some time ago I noticed that Fedora is building all their packages with very minimal locales installed quite successfully. In fedora, if there is the occasional packaging testsuite that requires a specific locale being installed, Fedora just uses "BuildRequires: glibc-langpack-$lang".

openSUSE currently doesn't have this granularity. we have glibc-locale-base, which covers C and en_US locale and we have glibc-locale which covers everything else.

The vast majority of packages are building and testing fine in the default C.utf8/en_US.utf8 call. By switching to glibc-locale-base, we're saving about 227 MB of data to pull and install on every build. That's like 2TB of data less written for a full distro build. if you're behind crappy internet like a german citizen would be, then you love the speedup.

Any comments, concerns? Yes I'm aware a few packages will fail to build. I've done a full rebuild of :Ring:0-Bootstrap/1-Minimalx and had about 10 build failures, and submitted already fixes for all of them.

Thank you for working on this. Leaner build env is a welcome change in almost every situation. As you're also busy on eventual fallouts, I am not worried at all to get this settled rather quickly Cheers, Dominique

815

Age (days ago)

819

Last active (days ago)

List overview

Download

12 comments

7 participants

participants (7)

Christian Boltz
Cristian Rodríguez
Dirk Müller
Dominique Leuenberger / DimStar
Guillaume Gardet
Jan Engelhardt
Neal Gompa

RFC: changing default build environment to use only glibc-locale-base

tags

participants (7)