From sid packages migrate to the testing release after they pass sanity (10 days delay + all dependencies are satisfiable) into testing. The migration is no more than change in the archive metadata

Hello there I'm Ubuntu Contributing Developer and I maintain a few packages in Debian as well. I would find it extremly useful to have filtered Universe in OSB. Information about all packages in Debian and Ubuntu is available as an sql database published in Ultimate Debian Database [1] which I have access to from alioth.debian.org. Futhermore all information about packages is also available with launchpadlib API. There are a few scripts on ubuntuwire QA website [2] which provide valuable statistics about Ubuntu & Debian archives and their comparison. Using this information it is quite easy to create any filtered responses as to which packages you want. Debian & Ubuntu position with respect to legallity and "freedom" are different. Debian is ultimatly bound by social contract [3] and The Debian Free Software Guidelines (DFSG) are part of it. This is how debian archive is split: main is free, contrib is free but depends on packages from non-free, and non-free is not part of the Debian OS at all and is purly volunteer run project. Note though there are still restrictions applied to non-free (i'm not involved with that). Ubuntu on the other hand takes a historically slightly different approach. You can view full break-down of per-component policy here [4]. In general main is inspected and free, restricted is inspected & is not free (more than debian non-free), universe may-be inspected by debian or might come from anywhere and should be free, multiverse is non-free (debian non-free). Please note that Ubuntu is blacklisting some packages from Debian main as well due to various reasons (techical, political, platform decisions etc.) If you are filtering debian main or ubuntu main for legal releasons please file bugs in Debian BTS and or Launchpad (both provide Security options to file privates bugs non in public view). If there are any concerns with those two components we (as in Debian/Ubuntu sides) need to sort them out on our side. In debian these type of bugs are considered Release-Critical. (Debian before release does a general resolution across all of it developers with respect to some kernel binary blobs that are distributed in main so those packages are out of the question). Lucas (Debian & Ubuntu Developer specialiases in large scale QA in both Debian & Ubuntu) has done a very insightful presentation at Fosdem 2010 [5]. As can be seen from that presentation 89% of packages come to ubuntu from Debian (patched by ubuntu or not). 11% are same packages as in debian but newer version in ubuntu (4% is about who did the packaging e.g. who wrote the original spec file in RPM speak) The proposal here is to make OSB import filters maintainable and blacklist from ubuntu following (I'm assuming you are not rebuilding ubuntu archives and you are mirroring them): Using udd [1] 1) Block everything that is blocked in Debian 2) Block everything that is not found in debian (will take the langpacks away as well which are pointless for OBS) This will create almost perfect filtered view of Ubuntu main & universe for OSB purposes. Now "supported" is a broad term in ubuntu cause it can apply to packages in universe as well. You can take another approach by using Seed Management [6] (Also known as Archive Reorganisation) a few cycles ago Ubuntu realised it is quite a hassle to manage all Ubuntu derivatives using just 4 components. (This kind of resembles how OBS is used were Factory is made up out of few blessed projects on OSB eg. Gnome:Factory etc.) So seeds have been actively used for many releases now. And I believe it is quite easy to create a seed which only pulls supported packages (quite a few are in universe because they are for derivatives e.g. Xubuntu and etc.) and blacklist those packages you don't want using the seed. Process it with [7] and you will have self-contained list of everything you are ok with to mirror. I would want to see the current blacklist & rules / scripts / software used to manage filtering Debian/Ubuntu mirrors. And i wouldn't mind to keep this upto date. Now the questions about the rolling releases: ubuntu+1, debian testing & debian unstable. I don't see how they are different from the Factory and I would like also to cover the unbiguity about ubuntu-proposed, ubuntu-security, ubuntu-backports, ubuntu-updates and similarly debian-security, debian-updates and debian backports. Backports, proposed-updates & volatile are cans of worms and there is no point to discuss them for mirroring to OSB =) During ubuntu development you have only one component open which is just codename eg currently: sid, maverick. First up Debian: Uploads are done directly into that and it contains the full archive. the binaries are the same and stay in the same directory and are not moved anywhere. Similarly when the big release comes the only change is that symlink directory called "stable" starts to points to a new codename ("stable" symlink will be changed from lenny to squeeze when it is ready). Similar thing happens with ubuntu+1 except that we don't have testing and we don't bother with symlinks =) we simply propogate metadata saying that new codename is release and upgrade software changes repository config to point e.g. from karmic to lucid. After the release the actuall lucid, lenny etc metadata, binaries & source are never changed and are frozen in stone. In order to provide stable release updates we have adjasted repositories which do not contain full release. These are <codename>-updates, <codename>-security these again have main, restricte, universe, multiverse components. When there is a need for update an upload is done into those. Packages are never removed from those only added. So although new updates are pushed there, they are semi-frozen and go though regious approval periods to make sure they don't break stable release. People are expected to have those enabled cuase otherwise they will not get any updates. So in order to support ubuntu/debian you should mirror the packages you want and regenerate metadata for release. PPA's on launchpad allow you to enable -security & -updates as well but this is not required at all. And on PPA's you can disable components as well (main, main+restricted, main+restriced+universe). Due to how debian archieve is organised you cannot have a same package version which is in fact a different binary across all of your releases everywhere. Furthermore we encode all API/ABI in package names & symbol tables such that it is impossible to install inconsistent state. So for example libfoo in debian would create binary packages: libfoo1, libfoo-dev. And then you will have package baz which will build-depened on libfoo-dev and gain a runtime dependency on libfoo1 (== exact version). When ABI is broken a new upload is done first of libfoo. At this point if you install baz you will get the old libfoo1 version because this is the version you strongly depend on. Then baz is uploaded again without any source change to get rebuild against new upload of libfoo at this point baz version will get incremented (buildX or binNMUX in ubuntu & debian respectivly) after this point users will be able to upgrade simultaniously to new baz & libfoo1. When API is broken (so-name bump) the binary package is renamed to libfoo2 and again you will have to reupload all dependencies. What this means is that archive as a whole is inconsistent at times and non-releasable because users cannot install everything simultaniously. It also means that you can remove binary package from the archive only when all binary package dependencies have been rebuild against newer version. But this is great for a build service because you cannot by-design produce inconsistent ABI/API package! You can only ever produce packages which will be uninstallable =) or you will fail to satisfy your build dependencies and fail the build at the very start. Ubuntu and Debian mirrors are using a two stage sync.[8] First sync all binary packages (takes a while), then sync meta-data & delete unreferenced binary/source packages. During first sync there is no change to the users cause they still see old-metadata. If the user manages to catch mirror during meta-data update gpg signatures on all the files will fail so it will be impossible to download nor install those packages via apt and after the meta-data is synced it is safe to delete unreferenced package cause noone will be able to request those off your mirror. For the purpose of OSB you will just blacklists the folders you don't want (using any method you like to generate that blacklist), do an rsync, regenerate metadata, and then delete unreferenced packages. There are no API/ABI inconsistencies some packages in the current releases of debian & ubuntu have been build many years ago and still work =) because they just depend on very api/abi stable libraries and don't have any known vonlubirities. As an added bonus in debian only arch specific packages get rebuilds for ABI bumps so you don't need to mirror all (noarch) uselessly =) So the reason "we don't know when to mirror rolling debian/ubuntu release" is absurd to me since ubuntu and debian (separatly) only have one repository for all of their releases & separate metadata generated against that repository to virtually create testing, unstable, maverick, lucid, lucid-security etc.... And debian binary package format hasn't ever changed yet it is ar archive of two tar.gz tarballs. Lenny and up, Hardy and up do support bz2 & lzma compressed tarballs inside the deb package & sid/squeeze recently started to support xz but this has not yet been enabled due to various stability reasons. Debian is not intending to do switch of their repository just yet and it acutally don't make sence for us cause we don't want to recompress the whole archive. If there any questions with respect to API/ABI stability in the "rolling" releases and how to filter the debian/ubuntu archives please ask I'll be glad to help & resolve any of those issues and see universe making debute on OSB =) [1] http://udd.debian.org/ [2] http://qa.ubuntuwire.net [3] http://www.debian.org/social_contract [4] http://www.ubuntu.com/community/ubuntustory/components [5] http://www.loria.fr/~lnussbau/files/fosdem2010-debian-ubuntu.pdf [6] https://wiki.ubuntu.com/SeedManagement [7] https://wiki.ubuntu.com/Germinate [8] https://wiki.ubuntu.com/Mirrors/Scripts [9] that will mean that your archive is no-longer verifying using ubuntu/debian-archive keys we will just trust you =) ps. When debian/ubuntu does "archive rebuilds" they are actually done separatly from the real archive and the results are used to identify the problems and prepare new uploads to the canonical archive if there is need. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org