[opensuse-factory] 'xz' (LZMA) support by default on our build environments...

newer
[opensuse-factory] [devel-editors]...

Nelson Marques

23 Dec 2011 23 Dec '11

14:29

Dear people, The GNOME Project, one of the biggest projects in openSUSE is throwing out the source code in .xz (LZMA). This is quite fun and there's obvious benefits on it, the problem that rises from this is the following: * Our build environments doesn't support .xz sources; * We're required to add "BuildRequires: xz" to the spec files; * Some projects require us to add comments to the "BR xz" entries refering to bnc#697467; * Some projects require us to add the same stuff in the changelog entries; In the last times I've seen a growing demonstration of attitudes in the scope of "We dont give a flying fuck about you or your needs" (that's how I see it). The Project Management is growing arrogant and following a potential hidden agenda and ignoring everything that doesn't sound nice to them. While we move forward (or some at least) into dangerous fields like systemd, btrfs, the proposed changes from Fedora to the filesystem structure and friends... the stuff that we really could use to improve productivity and our work is being left behind. So I bring this tiny issue to your attention, so that any kind soul can provide a technical explanation on why we can have 'xz' supported on our build environments... and why we're required to insert dumbass comments that no one can explain. If the Project Leadership is unable to answer with a technical explanation for this, you should seriously consider resigning and opening your spot to someone who has some balls to change things up and at least provide a reasonable explanation for this.... at the risk of loosing every independent contributor that still donates some of their free time to openSUSE. NM PS: If you don't like the tone, live with it, I return in double whatever is granted to me. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Show replies by date

Michael Schroeder

23 Dec 23 Dec

14:50

On Fri, Dec 23, 2011 at 02:29:22PM +0000, Nelson Marques wrote: [rant deleted]

...

So I bring this tiny issue to your attention, so that any kind soul can provide a technical explanation on why we can have 'xz' supported on our build environments...

What's wrong with adding "BuildRequires: xz"? We're working hard to reduce the number of automatically installed packages by moving stuff that is only required by some packages into BuildRequires. We're not doing this to annoy people, We do it to reduce package build time.

...

and why we're required to insert dumbass comments that no one can explain.

There's a policy that changelog comments need to refer to a bug for maintenance updates, but this is the Factory mailing list, so I have no clue what you're talking about. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Nelson Marques

15:15

2011/12/23 Michael Schroeder :

...

On Fri, Dec 23, 2011 at 02:29:22PM +0000, Nelson Marques wrote:

[rant deleted]

...
So I bring this tiny issue to your attention, so that any kind soul can provide a technical explanation on why we can have 'xz' supported on our build environments...

What's wrong with adding "BuildRequires: xz"?

Nothing is wrong with it, it's just superfluous.

...

We're working hard to reduce the number of automatically installed packages by moving stuff that is only required by some packages into BuildRequires. We're not doing this to annoy people, We do it to reduce package build time.

Let me see if I understood this correctly: * I want to package foobar-1.0.tar.xz; (upstream releases only .xz) * I have to introduce: "BuildRequires: xz" * I have to waste time introducing references to why I have to include a "BuildRequires: xz" * I have to increase the size and the complexity of the spec file; * I have to spend extra time referencing more stuff in the changelog; In the end... 'xz' which you don't add to the build environment is still installed. Where is exactly the gain you try to point here? (besides the fact that you deliberally are increasing the maintenance time of my package) Please demonstrate how my package is built faster by not having xz installed on the build environment by default (but is installed still). Let me guess... for your own pleasure and to support idea you preach, I probably have still to spend more time unpacking and repacking the sources in bz2 or gz ? :) My brain is probably going through a meltdown, because all I can see in this is mainly: "increase the load on the packagers". And it's nice that now I have to reference a few incidents on bugzilla so I can produce comments to go with the buildrequirements that aren't really "Build Requirements" and should be part of the build environment... If upstream projects are moving to .xz (not just GNOME, but also many which have huge tarballs), what do we earn here ? Another thing that pops up is that this option will probably increase the load in other areas, for example network traffic because a 100Mb source in bz2 is most likely around 70Mb in .xz. I'm missing really something important here...

...

...
and why we're required to insert dumbass comments that no one can explain.

There's a policy that changelog comments need to refer to a bug for maintenance updates, but this is the Factory mailing list, so I have no clue what you're talking about.

...

Cheers, Michael.

-- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}

-- Nelson Marques /* http://www.marques.so nmo.marques@gmail.com */ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Michael Schroeder

15:27

On Fri, Dec 23, 2011 at 03:15:48PM +0000, Nelson Marques wrote:

...

Let me see if I understood this correctly:

* I want to package foobar-1.0.tar.xz; (upstream releases only .xz) * I have to introduce: "BuildRequires: xz"

Yes.

...

* I have to waste time introducing references to why I have to include a "BuildRequires: xz"

No.

...

* I have to increase the size and the complexity of the spec file;

You seriously think that BuildRequires is complex?

...

* I have to spend extra time referencing more stuff in the changelog;

No.

...

In the end... 'xz' which you don't add to the build environment is still installed. Where is exactly the gain you try to point here? (besides the fact that you deliberally are increasing the maintenance time of my package)

Please demonstrate how my package is built faster by not having xz installed on the build environment by default (but is installed still).

This is not about your package, other packages that don't need xz will be built faster. With your logic, we should also install Xorg, an ocaml compiler, a jave compiler and so on, because some package might need it.

...

Let me guess... for your own pleasure and to support idea you preach, I probably have still to spend more time unpacking and repacking the sources in bz2 or gz ? :)

No! Do *not* repackage the sources! Just add that BuildRequires line.

...

My brain is probably going through a meltdown, because all I can see in this is mainly: "increase the load on the packagers". And it's nice that now I have to reference a few incidents on bugzilla so I can produce comments to go with the buildrequirements that aren't really "Build Requirements" and should be part of the build environment...

Wtf? As I said, that bugzilla reference is for *released* products, not Factory. You're talking about a new package or a new version.

...

If upstream projects are moving to .xz (not just GNOME, but also many which have huge tarballs), what do we earn here ? Another thing that pops up is that this option will probably increase the load in other areas, for example network traffic because a 100Mb source in bz2 is most likely around 70Mb in .xz.

No, 'cause you should not repackage. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Nelson Marques

15:43

2011/12/23 Michael Schroeder :

...

On Fri, Dec 23, 2011 at 03:15:48PM +0000, Nelson Marques wrote:

...
Let me see if I understood this correctly:

* I want to package foobar-1.0.tar.xz; (upstream releases only .xz) * I have to introduce: "BuildRequires: xz"

Yes.

Sure.

...

...
* I have to waste time introducing references to why I have to include a "BuildRequires: xz"

No.

Next time I'm asked to do so, package gets orphaned right on that moment.

...

...
* I have to increase the size and the complexity of the spec file;

You seriously think that BuildRequires is complex?

Yes, it should be a part of the build environment. It's not a build dependency.

...

...
* I have to spend extra time referencing more stuff in the changelog;

No.

Next time I don't get a package accepted because of it, it gets orphaned right on the very same moment.

...

...
In the end... 'xz' which you don't add to the build environment is still installed. Where is exactly the gain you try to point here? (besides the fact that you deliberally are increasing the maintenance time of my package)

Please demonstrate how my package is built faster by not having xz installed on the build environment by default (but is installed still).

This is not about your package, other packages that don't need xz will be built faster. With your logic, we should also install Xorg, an ocaml compiler, a jave compiler and so on, because some package might need it.

You can give all the examples you want, you are still failing to realize XZ is used to unpack the source code and not to build the application. That's probably why it _should_ be a part of the build environment. Please, whats the difference between gzip, bunzip and xz ? Why are they treated differently ?

...

...
Let me guess... for your own pleasure and to support idea you preach, I probably have still to spend more time unpacking and repacking the sources in bz2 or gz ? :)

No! Do *not* repackage the sources! Just add that BuildRequires line.

Why? using your logic, if I repackage the sources with bunzip, it gets built faster! Afterall you don't install .xz and you save precious time there :)

...

...
My brain is probably going through a meltdown, because all I can see in this is mainly: "increase the load on the packagers". And it's nice that now I have to reference a few incidents on bugzilla so I can produce comments to go with the buildrequirements that aren't really "Build Requirements" and should be part of the build environment...

Wtf? As I said, that bugzilla reference is for *released* products, not Factory. You're talking about a new package or a new version.

I'm talking about packages and development repo's, updates mainly. I'm a puppet maintainer, I'm allowed to maintain packages on low profile repositories and hammer them in the way I want, but I'm not allowed to maintain the same packages on the official development repository, so I might get my packages declined. Considering that everyone is placing good efforts in killing volunteer contributions, I can see a good ammount of packages becoming orphaned :)

...

...
If upstream projects are moving to .xz (not just GNOME, but also many which have huge tarballs), what do we earn here ? Another thing that pops up is that this option will probably increase the load in other areas, for example network traffic because a 100Mb source in bz2 is most likely around 70Mb in .xz.

No, 'cause you should not repackage.

Good. Sounds like a Mexican standout :) This generates orphaned packages.

...

Cheers, Michael.

-- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}

Andreas Jaeger

20:15

On 12/23/2011 04:43 PM, Nelson Marques wrote:

...

2011/12/23 Michael Schroeder:

...
On Fri, Dec 23, 2011 at 03:15:48PM +0000, Nelson Marques wrote:

...
Let me see if I understood this correctly:

* I want to package foobar-1.0.tar.xz; (upstream releases only .xz) * I have to introduce: "BuildRequires: xz"

Yes.

Sure.

...
...
* I have to waste time introducing references to why I have to include a "BuildRequires: xz"

No.

Next time I'm asked to do so, package gets orphaned right on that moment.

No, you reference the email by mls (use http://lists..., and resubmit - and if that does not work, answer to the email and note the submitrequest and ask for help. Not everybody reads every email and remembers it, we're friends here and if not, let's ask friends to help out ;) Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Nelson Marques

20:45

2011/12/23 Andreas Jaeger :

...

On 12/23/2011 04:43 PM, Nelson Marques wrote:

...
2011/12/23 Michael Schroeder:

...
On Fri, Dec 23, 2011 at 03:15:48PM +0000, Nelson Marques wrote:

...
Let me see if I understood this correctly:

* I want to package foobar-1.0.tar.xz; (upstream releases only .xz) * I have to introduce: "BuildRequires: xz"

Yes.

Sure.

...
...
* I have to waste time introducing references to why I have to include a "BuildRequires: xz"

No.

Next time I'm asked to do so, package gets orphaned right on that moment.

No, you reference the email by mls (use http://lists..., and resubmit - and if that does not work, answer to the email and note the submitrequest and ask for help.

Andreas (and all), No, like I said before, the package gets orphaned. The "management" is ignoring us for too long, it's time they actually listen to people, if they don't care, then why I should I ? They turn their backs on us, we turn our backs to openSUSE. That's how it works. NM

...

Not everybody reads every email and remembers it, we're friends here and if not, let's ask friends to help out ;)

Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Rüdiger Meier

16:35

On Friday 23 December 2011, Michael Schroeder wrote:

...

On Fri, Dec 23, 2011 at 02:29:22PM +0000, Nelson Marques wrote:

[rant deleted]

...
So I bring this tiny issue to your attention, so that any kind soul can provide a technical explanation on why we can have 'xz' supported on our build environments...

What's wrong with adding "BuildRequires: xz"?

Beside Nelson's arguments I also think that xz becomes as essential as gzip, bzip2, tar or even coreutils. Trivial BuildRequires are useless. Should we require even "patch" to let %patch work? Why doing this for %setup which is even more often used than %patch? IMO the tar package itself should Require xz (and gzip, bzip2!) because it's not very useful without them. On the other hand tar requires "info" - that's really unbalanced. BTW it's also really pity that this feature got rejected. https://features.opensuse.org/312439 cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Cristian Rodríguez

24 Dec 24 Dec

17:20

On 23/12/11 11:29, Nelson Marques wrote:

...

Dear people,

The GNOME Project, one of the biggest projects in openSUSE is throwing out the source code in .xz (LZMA). This is quite fun and there's obvious benefits on it, the problem that rises from this is the following:

* Our build environments doesn't support .xz sources; * We're required to add "BuildRequires: xz" to the spec files; * Some projects require us to add comments to the "BR xz" entries refering to bnc#697467; * Some projects require us to add the same stuff in the changelog entries;

The solution to this issue is to simple change the default "tar" package in the build system by bsdtar which supports all formats, including zip, xz, bzip etc.. using a library. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Anders Johansson

17:49

On Saturday 24 December 2011 14:20:24 Cristian > The solution to this issue is to simple change the default "tar" package

...

in the build system by bsdtar which supports all formats, including zip, xz, bzip etc.. using a library.

Our GNU tar already does that. In 12.1 it supports xz compressed tar files out of the box. No need to replace anything. Anders -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Cristian Rodríguez

18:00

On 24/12/11 14:49, Anders Johansson wrote:

...

On Saturday 24 December 2011 14:20:24 Cristian> The solution to this issue is to simple change the default "tar" package

...
in the build system by bsdtar which supports all formats, including zip, xz, bzip etc.. using a library.

Our GNU tar already does that. In 12.1 it supports xz compressed tar files out of the box. No need to replace anything.

Not the same way, gnu tar forks the XZ, gzip, bzip binaries (hence the need of BuildRequires: xz), bsdtar uses libarchive instead. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Nelson Marques

20:39

Hi, This sounds wonderful[1]. Very good news and an excellent christmas gift... I would love to see this happening. NM [1] - http://code.google.com/p/libarchive/ 2011/12/24 Cristian Rodríguez :

...

On 24/12/11 14:49, Anders Johansson wrote:

...
On Saturday 24 December 2011 14:20:24 Cristian> The solution to this issue is to simple change the default "tar" package

...
in the build system by bsdtar which supports all formats, including zip, xz, bzip etc.. using a library.

Our GNU tar already does that. In 12.1 it supports xz compressed tar files out of the box. No need to replace anything.

Not the same way, gnu tar forks the XZ, gzip, bzip binaries (hence the need of BuildRequires: xz), bsdtar uses libarchive instead.

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Cristian Rodríguez

22:11

On 24/12/11 17:39, Nelson Marques wrote:

...

Hi,

This sounds wonderful[1]. Very good news and an excellent christmas gift... I would love to see this happening.

NM

It is not that wonderful actually, it will require adding extra packages to the base installation anyway, defeating the original purpose that is having the less as possible packages in the base buildroot. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Claudio Freire

22:52

On Sat, Dec 24, 2011 at 7:11 PM, Cristian Rodríguez wrote:

...

It is not that wonderful actually, it will require adding extra packages to the base installation anyway, defeating the original purpose that is having the less as possible packages in the base buildroot.

Ok, but if a big portion of upstream projects start shipping xz tarballs, is one extra package that much? That could be the case if many apps follow gnome 3's suit, which seems to be the case already for many gnome core apps. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Nelson Marques

25 Dec 25 Dec

00:48

2011/12/24 Claudio Freire :

...

On Sat, Dec 24, 2011 at 7:11 PM, Cristian Rodríguez wrote:

...
It is not that wonderful actually, it will require adding extra packages to the base installation anyway, defeating the original purpose that is having the less as possible packages in the base buildroot.

Ok, but if a big portion of upstream projects start shipping xz tarballs, is one extra package that much?

Cristian, For example, some days ago we start rebuilding over 40 packages for MATE (the Mint featured desktop alternative), those packages once more came in .xz. But there might be another potential workaround... Is it possible to configure the build environment base packages in OBS for a project? That way probably would probably allow people to tweak their repo's for extended 'functionality'. Personally I think it's somehow bad to reduce too much the build environment... and it depends a bit on which kind of packages you are building, normal releases don't need much stuff around, while if you're building CVS/git sources it takes a few extra packages. Cutting down too much stuff isn't actually good, depends too much on what kind stuff people work with. Sure it will improve some projects, but will also make others far more dificult.

...

That could be the case if many apps follow gnome 3's suit, which seems to be the case already for many gnome core apps.

GNOME itself is big enough to justify it, that's my opinion, but they are not the only ones. All Mate development packages are in .xz for example (the releases are also in bz2). If you have documentation (I'm not even sure if its possible) to use project metadata to define extra packages for a project base build environment, will work for me for most situations. NM PS: but I did liked your suggestion, bsdtar and libarchive seem nice for this kinda stuff, even for normal user usage as it offers pretty much all the functionality I require for archive management.

...

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Cristian Rodríguez

03:16

On 24/12/11 21:48, Nelson Marques wrote:

...

2011/12/24 Claudio Freire:

...
On Sat, Dec 24, 2011 at 7:11 PM, Cristian Rodríguez wrote:

...
It is not that wonderful actually, it will require adding extra packages to the base installation anyway, defeating the original purpose that is having the less as possible packages in the base buildroot.

Ok, but if a big portion of upstream projects start shipping xz tarballs, is one extra package that much?

Cristian,

For example, some days ago we start rebuilding over 40 packages for MATE (the Mint featured desktop alternative), those packages once more came in .xz.

But there might be another potential workaround... Is it possible to configure the build environment base packages in OBS for a project? That way probably would probably allow people to tweak their repo's for extended 'functionality'.

Yes, osc meta -e prjconf your:project add Support: xz that's all. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Linda Walsh

31 Dec 31 Dec

02:57

New subject: [opensuse-factory] Re: 'xz' (LZMA) support by default on our build environments...

Cristian Rodríguez wrote:

...

On 24/12/11 21:48, Nelson Marques wrote:

...
2011/12/24 Claudio Freire:

...
On Sat, Dec 24, 2011 at 7:11 PM, Cristian Rodríguez wrote:

...
It is not that wonderful actually, it will require adding extra packages to the base installation anyway, defeating the original purpose that is having the less as possible packages in the base buildroot.

Ok, but if a big portion of upstream projects start shipping xz tarballs, is one extra package that much?

Cristian,

For example, some days ago we start rebuilding over 40 packages for MATE (the Mint featured desktop alternative), those packages once more came in .xz.

---- Can I make a suggestion? How about adding a package that removes others? Like if you 'add' 7z, you can remove, gzip, zip, bzip2, uncab, unrar, lzma/unlzma, and xz... it also unpacks tar, cpi, rpm and deb format as well as several OBJ formats... For compression, it handles (compresses in:) zip, gzip bzip2, (and xz/7z)... And it does a better job of zip compression than zip! (it can also compress in any of those formats in parallel, but only handles bzip2 for parallel decompression... I was just thinking about how simpler the 'lessopen.sh' plugin could be and how much more it could handle automatically using 1 program as it's filter. ... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Joerg.Schilling＠fokus.fraunhofer.de

13:59

New subject: [opensuse-factory] Re: 'xz' (LZMA) support by default on our build environments...

Linda Walsh wrote:

...

Can I make a suggestion?

How about adding a package that removes others?

Like if you 'add' 7z, you can remove, gzip, zip, bzip2, uncab, unrar, lzma/unlzma, and xz...

it also unpacks tar, cpi, rpm and deb format as well as several OBJ formats...

For compression, it handles (compresses in:) zip, gzip bzip2, (and xz/7z)...

And it does a better job of zip compression than zip! (it can also compress in any of those formats in parallel, but only handles bzip2 for parallel decompression...

You did not really try that..... 7z only supports some ancient tar formats but not POSIX.1-2001 and it does not behave correctly for uncompressing data as it does not implement the compress(1) CLI. xz e.g. was written in order to get a compliant CLI for the 7z compression method. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

4498

Age (days ago)

4506

Last active (days ago)

List overview

Download

17 comments

9 participants

participants (9)

Anders Johansson
Andreas Jaeger
Claudio Freire
Cristian Rodríguez
Joerg.Schilling＠fokus.fraunhofer.de
Linda Walsh
Michael Schroeder
Nelson Marques
Rüdiger Meier