Hi, Robert Schiele schrieb:

But what was the _reason_ for the debuginfo split? Just that some people wanted to have it without having a reason?

this was my proposal and is therefore my "fault". https://bugzilla.novell.com/show_bug.cgi?id=197823 So I'd like to defend myself. There are currently more than 6000 binary packages in Factory-x86_64, 2000 of which are debuginfo packages. The noarch packages are less than 1000. I guess that the situation is nearly the same on all architectures. So there are about 7000 binary packages per architecture, 2000 of which are debuginfo packages. Someone has to parse all this stuff. I mean, the metadata. It is well known that zypp parses the repository metadata slowly. It has already become faster and it will become even better, but it's still slow. And it's not just zypp. Yum, with the new(!) C metadata parser written by Tambet Ingo, needs half a minute to parse primary.xml and again half a minute to parse filelists.xml on my laptop. I don't even want to know how slow it would be with the old python parser. While parsing, it shows me that it parses the metadata for about 22000 packages, probably because it has to parse the metadata for all architectures. Among the 22000 packages there are 6000 debuginfo packages, 2000 per architecture. My idea was that this can be reduced between 1/3 and 1/4 "for free" by separating the debuginfo packages. It is of course not entirely "for free" because it is less convenient for those people who want to have the debuginfo packages. But these packages are a rather specialized use case. There are people who need them every day and there are people who never need them, maybe there are even people who are confused by them. For Factory, the ratio of people who need them is probably larger than for a released version. The proposal was primarily intended for the released versions. OK, now we have it for Factory, too. The debuginfo packages contribute quite a lot to the repository metadata because they contain the complete source code, every single file unpacked into /usr/src/debug. For a rather small library like zlib, these are 26 files while the main and devel packages just have 7 files each. And there are larger packages with more source files and still just 7 binary files, where the ratio is even worse. Andreas Hanke --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Fri, Aug 25, 2006 at 05:54:18AM +0200, Andreas Hanke wrote:

Someone has to parse all this stuff. I mean, the metadata. It is well known that zypp parses the repository metadata slowly. It has already become faster and it will become even better, but it's still slow.

And it's not just zypp. Yum, with the new(!) C metadata parser written by Tambet Ingo, needs half a minute to parse primary.xml and again half a minute to parse filelists.xml on my laptop. I don't even want to know how slow it would be with the old python parser.

OK, so you are proposing a *workaround* for a known and very severe problem. Especially with factory, we should *not* concentrate on workaround but on *fixes*! So as long as factory is a development branch, this *should not* be done. ciao Joerg -- Joerg Mayer We are stuck with technology when what we really want is just stuff that works. Some say that should read Microsoft instead of technology. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Fri, Aug 25, 2006 at 10:45:56AM +0200, Adrian Schroeter wrote:

Basicaly two reasons:

1. Mirrors can skip the debuginfo packages, without an exclude rule and without to "break" the repository meta data.

2. The installers have less meta data to handle by default (when you ignore the -debug repo), this let them take less memory, faster downloads of the meta data and faster solving.

Thanks. I'd appreciate to see short explanations like this one in the first place when making announcements like this. Just saying "on popular request" did not sound as if the decission was really based on a reason. Robert -- Robert Schiele Tel.: +49-621-181-2214 Dipl.-Wirtsch.informatiker mailto:rschiele@uni-mannheim.de "Quidquid latine dictum sit, altum sonatur."

Hi, Joerg Mayer schrieb:

OK, so you are proposing a *workaround* for a known and very severe problem. Especially with factory, we should *not* concentrate on workaround but on *fixes*! So as long as factory is a development branch, this *should not* be done.

Joerg, this is a valid point if it were a workaround, but it is not entirely a workaround. Do you know any other distro that has such a huge package base as openSUSE and uses rpm-md? I don't. Before making the proposal, I looked at Fedora's repository layout and found out the following: - Fedora Core has significantly fewer packages than openSUSE. - Fedora already splits the debuginfo packages out, into a separate directory that doesn't have any repository metadata at all. - Fedora also separates the repositories per architecture even though rpm-md supports multiarch repositories very well. - Fedora splits the source packages into a separate repository from the binary packages. In numbers. openSUSE: 4000 "real" i586 binary packages. 2000 i586 debuginfo packages. 4000 "real" x86_64 binary packages. 2000 x86_64 debuginfo packages. 4000 "real" ppc binary packages. 2000 ppc debuginfo packages. 1000 noarch packages. 3000 source packages. ---- 22000 total packages in a SINGLE repository. Fedora: 2200 packages in the repository most people are interested in (i586 binary + noarch, no debuginfo, no source, no other architectures). 22000/2200 is a factor of 10! This makes me seriously doubt that rpm-md is designed or even suitable for such huge repositories. It's not surprising that parsing this beast is slow, even with a fast parser. It also makes me doubt that improving the parser is the only way of approaching the problem. So I thought how to reduce the number of packages: - Separate repositories per architecture - not possible because SUSE repositories have always been multiarch. - Separate repositories for source packages - bad idea IMHO. - Reducing the number of packages - not possible, people want to have more software and not less. The debuginfo packages sounded like a reasonable candidate to me because their number is always proportional to the number of binary packages. If we get a new architecture like ia64, we get more debuginfo packages as well, which means we can also save proportionally by splitting them from the rest. Note also that we already have performance workarounds. Using the old susetags metadata instead of rpm-md during initial installation is one of them. Andreas Hanke --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

* Andreas Hanke [Aug 25. 2006 15:24]: [...]

22000 total packages in a SINGLE repository.

Fedora:

2200 packages in the repository most people are interested in (i586 binary + noarch, no debuginfo, no source, no other architectures).

22000/2200 is a factor of 10!

Yes, this is one of the problems we're facing.

This makes me seriously doubt that rpm-md is designed or even suitable for such huge repositories. It's not surprising that parsing this beast is slow, even with a fast parser. It also makes me doubt that improving the parser is the only way of approaching the problem.

So I thought how to reduce the number of packages:

- Separate repositories per architecture - not possible because SUSE repositories have always been multiarch.

It not impossible, but needs extra work. Currently its also nice to publish only one repo URL without the need to distinguish between different architectures.

- Separate repositories for source packages - bad idea IMHO.

Why do you think this is a bad idea ?

- Reducing the number of packages - not possible, people want to have more software and not less.

Reducing the number of packages _per_ repository is easily possible. If you want more software, add more repositories. The OpenSUSE build service is supposed to address this in the future. What we need is some kind of 'meta repository' which points to other repositories. Its planned but we do not know if its doable in the OpenSUSE 10.2 timeframe. Klaus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

* Andreas Hanke [Aug 25. 2006 16:24]:

Hi,

Klaus Kaempf schrieb:

...

...

- Separate repositories per architecture - not possible because SUSE repositories have always been multiarch.

It not impossible, but needs extra work. Currently its also nice to publish only one repo URL without the need to distinguish between different architectures.

And there would have to be a solution for biarch - because x86_64 needs to know about the i586 packages.

All of them ? Probably not. But it might be too much work to calculate needed (like 32 bit libraries) one.

Are the metadata of source packages faster or slower or equal to parse? Do they have equally verbose dependency information?

They are faster to parse because they usually have no dependency information. Klaus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Hi, On Fri, 25 Aug 2006, Andreas Hanke wrote:

Andreas Hanke schrieb:

...

...

...

- Separate repositories for source packages - bad idea IMHO.

Why do you think this is a bad idea ?

Because they would be harder to find, resulting in fake GPL violation discussions on this list :-(

Another point to consider: If it becomes possible to mirror the distribution without the source RPMs, someone will do it; and if someone does it, people will come back here and ask why there are just binaries on a mirror and not the sources.

Brute-forcing mirrors to have the source RPMs by tying them together with the binary RPMs is a desirable effect IMHO.

Serving fools just creates a new fool. So this should not be an "originary" aspect. Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

jdd schrieb:

but why couldn't you have _one_ repository and _several_ metadata files?

That's not possible with rpm-md. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org