Mailinglist Archive: opensuse-buildservice (344 mails)

[Dmitrijs Ledkovs <dmitrij.ledkov@xxxxxxxxxx>]

Hello there

I'm Ubuntu Contributing Developer and I maintain a few packages in
Debian as well.

I would find it extremly useful to have filtered Universe in OSB.

Information about all packages in Debian and Ubuntu is available as an
sql database published in Ultimate Debian Database [1] which I have
access to from alioth.debian.org. Futhermore all information about
packages is also available with launchpadlib API.

There are a few scripts on ubuntuwire QA website [2] which provide
valuable statistics about Ubuntu & Debian archives and their
comparison. Using this information it is quite easy to create any
filtered responses as to which packages you want.

Debian & Ubuntu position with respect to legallity and "freedom" are different.

Debian is ultimatly bound by social contract [3] and The Debian Free
Software Guidelines (DFSG) are part of it.

This is how debian archive is split: main is free, contrib is free but
depends on packages from non-free, and non-free is not part of the
Debian OS at all and is purly volunteer run project. Note though there
are still restrictions applied to non-free (i'm not involved with
that).

Ubuntu on the other hand takes a historically slightly different
approach. You can view full break-down of per-component policy here
[4]. In general main is inspected and free, restricted is inspected &
is not free (more than debian non-free), universe may-be inspected by
debian or might come from anywhere and should be free, multiverse is
non-free (debian non-free). Please note that Ubuntu is blacklisting
some packages from Debian main as well due to various reasons
(techical, political, platform decisions etc.)

If you are filtering debian main or ubuntu main for legal releasons
please file bugs in Debian BTS and or Launchpad (both provide Security
options to file privates bugs non in public view). If there are any
concerns with those two components we (as in Debian/Ubuntu sides) need
to sort them out on our side. In debian these type of bugs are
considered Release-Critical. (Debian before release does a general
resolution across all of it developers with respect to some kernel
binary blobs that are distributed in main so those packages are out of
the question).

Lucas (Debian & Ubuntu Developer specialiases in large scale QA in
both Debian & Ubuntu) has done a very insightful presentation at
Fosdem 2010 [5]. As can be seen from that presentation 89% of packages
come to ubuntu from Debian (patched by ubuntu or not). 11% are same
packages as in debian but newer version in ubuntu (4% is about who did
the packaging e.g. who wrote the original spec file in RPM speak)

The proposal here is to make OSB import filters maintainable and
blacklist from ubuntu following (I'm assuming you are not rebuilding
ubuntu archives and you are mirroring them):

Using udd [1]

1) Block everything that is blocked in Debian
2) Block everything that is not found in debian (will take the
langpacks away as well which are pointless for OBS)

This will create almost perfect filtered view of Ubuntu main &
universe for OSB purposes.

Now "supported" is a broad term in ubuntu cause it can apply to
packages in universe as well.

You can take another approach by using Seed Management [6] (Also known
as Archive Reorganisation) a few cycles ago Ubuntu realised it is
quite a hassle to manage all Ubuntu derivatives using just 4
components. (This kind of resembles how OBS is used were Factory is
made up out of few blessed projects on OSB eg. Gnome:Factory etc.) So
seeds have been actively used for many releases now. And I believe it
is quite easy to create a seed which only pulls supported packages
(quite a few are in universe because they are for derivatives e.g.
Xubuntu and etc.) and blacklist those packages you don't want using
the seed. Process it with [7] and you will have self-contained list of
everything you are ok with to mirror.

I would want to see the current blacklist & rules / scripts / software
used to manage filtering Debian/Ubuntu mirrors. And i wouldn't mind to
keep this upto date.

Now the questions about the rolling releases: ubuntu+1, debian testing
& debian unstable. I don't see how they are different from the Factory
and I would like also to cover the unbiguity about ubuntu-proposed,
ubuntu-security, ubuntu-backports, ubuntu-updates and similarly
debian-security, debian-updates and debian backports.

Backports, proposed-updates & volatile are cans of worms and there is
no point to discuss them for mirroring to OSB =)

During ubuntu development you have only one component open which is
just codename eg currently: sid, maverick.

First up Debian:
Uploads are done directly into that and it contains the full archive.
> From sid packages migrate to the testing release after they pass
sanity (10 days delay + all dependencies are satisfiable) into
testing. The migration is no more than change in the archive metadata
the binaries are the same and stay in the same directory and are not
moved anywhere. Similarly when the big release comes the only change
is that symlink directory called "stable" starts to points to a new
codename ("stable" symlink will be changed from lenny to squeeze when
it is ready).

Similar thing happens with ubuntu+1 except that we don't have testing
and we don't bother with symlinks =) we simply propogate metadata
saying that new codename is release and upgrade software changes
repository config to point e.g. from karmic to lucid.

After the release the actuall lucid, lenny etc metadata, binaries &
source are never changed and are frozen in stone. In order to provide
stable release updates we have adjasted repositories which do not
contain full release. These are <codename>-updates,
<codename>-security these again have main, restricte, universe,
multiverse components. When there is a need for update an upload is
done into those. Packages are never removed from those only added. So
although new updates are pushed there, they are semi-frozen and go
though regious approval periods to make sure they don't break stable
release. People are expected to have those enabled cuase otherwise
they will not get any updates.

So in order to support ubuntu/debian you should mirror the packages
you want and regenerate metadata for release. PPA's on launchpad allow
you to enable -security & -updates as well but this is not required at
all. And on PPA's you can disable components as well (main,
main+restricted, main+restriced+universe).

Due to how debian archieve is organised you cannot have a same package
version which is in fact a different binary across all of your
releases everywhere. Furthermore we encode all API/ABI in package
names & symbol tables such that it is impossible to install
inconsistent state. So for example libfoo in debian would create
binary packages: libfoo1, libfoo-dev. And then you will have package
baz which will build-depened on libfoo-dev and gain a runtime
dependency on libfoo1 (== exact version). When ABI is broken a new
upload is done first of libfoo. At this point if you install baz you
will get the old libfoo1 version because this is the version you
strongly depend on. Then baz is uploaded again without any source
change to get rebuild against new upload of libfoo at this point baz
version will get incremented (buildX or binNMUX in ubuntu & debian
respectivly) after this point users will be able to upgrade
simultaniously to new baz & libfoo1. When API is broken (so-name bump)
the binary package is renamed to libfoo2 and again you will have to
reupload all dependencies.

What this means is that archive as a whole is inconsistent at times
and non-releasable because users cannot install everything
simultaniously. It also means that you can remove binary package from
the archive only when all binary package dependencies have been
rebuild against newer version.

But this is great for a build service because you cannot by-design
produce inconsistent ABI/API package! You can only ever produce
packages which will be uninstallable =) or you will fail to satisfy
your build dependencies and fail the build at the very start.

Ubuntu and Debian mirrors are using a two stage sync.[8] First sync
all binary packages (takes a while), then sync meta-data & delete
unreferenced binary/source packages. During first sync there is no
change to the users cause they still see old-metadata. If the user
manages to catch mirror during meta-data update gpg signatures on all
the files will fail so it will be impossible to download nor install
those packages via apt and after the meta-data is synced it is safe to
delete unreferenced package cause noone will be able to request those
off your mirror.

For the purpose of OSB you will just blacklists the folders you don't
want (using any method you like to generate that blacklist), do an
rsync, regenerate metadata, and then delete unreferenced packages.

There are no API/ABI inconsistencies some packages in the current
releases of debian & ubuntu have been build many years ago and still
work =) because they just depend on very api/abi stable libraries and
don't have any known vonlubirities. As an added bonus in debian only
arch specific packages get rebuilds for ABI bumps so you don't need to
mirror all (noarch) uselessly =)

So the reason "we don't know when to mirror rolling debian/ubuntu
release" is absurd to me since ubuntu and debian (separatly) only have
one repository for all of their releases & separate metadata generated
against that repository to virtually create testing, unstable,
maverick, lucid, lucid-security etc....

And debian binary package format hasn't ever changed yet it is ar
archive of two tar.gz tarballs. Lenny and up, Hardy and up do support
bz2 & lzma compressed tarballs inside the deb package & sid/squeeze
recently started to support xz but this has not yet been enabled due
to various stability reasons. Debian is not intending to do switch of
their repository just yet and it acutally don't make sence for us
cause we don't want to recompress the whole archive.

If there any questions with respect to API/ABI stability in the
"rolling" releases and how to filter the debian/ubuntu archives please
ask I'll be glad to help & resolve any of those issues and see
universe making debute on OSB =)

[1] http://udd.debian.org/
[2] http://qa.ubuntuwire.net
[3] http://www.debian.org/social_contract
[4] http://www.ubuntu.com/community/ubuntustory/components
[5] http://www.loria.fr/~lnussbau/files/fosdem2010-debian-ubuntu.pdf
[6] https://wiki.ubuntu.com/SeedManagement
[7] https://wiki.ubuntu.com/Germinate
[8] https://wiki.ubuntu.com/Mirrors/Scripts
[9] that will mean that your archive is no-longer verifying using
ubuntu/debian-archive keys we will just trust you =)

ps. When debian/ubuntu does "archive rebuilds" they are actually done
separatly from the real archive and the results are used to identify
the problems and prepare new uploads to the canonical archive if there
is need.
-- 
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse-buildservice+help@xxxxxxxxxxxx