[opensuse-kde] Faster KDE:Qt/KKFD rebuilds
Ho ho ho, since everybody working on KDE packaging have been good this year, you have been given faster rebuilds in KDE repos. Happy Christmas :). As some might have noticed when updating a KDE repo to a newer KDE SC version, it takes quite a while for it to rebuild, especially if Qt or kdelibs are triggered for rebuild. In fact, originally it looked roughly like this ('X' in KMail for non-proportional fonts is helpful; times are just rough measures and depend on OBS's mood, but they should match at least relatively): libqt4 1:00 (+1:00) -> libqt4-devel-doc 1:50 (+0:50) -> qt-creator 2:00 (+0:10) -> automoc4, soprano, strigi, phonon 2:20 (+0:20) -> kdelibs4 3:40 (+1:20) -> kdepimlibs4 4:00(+0:20) -> kdebase4-workspace 5:00 (+1:00) -> kdesdk4 5:30 (+0:30) -> kdevplatform 5:50 (+0:20) -> python-kde4 6:35 (+0:45) -> kdeutils4 6:55 (+0:20) -> kdepim4-runtime 5:20 (+0:20) -> kdepim4 6:30 (+1:10) -> kdegraphics4 4:05 (+0:25) -> koffice2 5:55 (+1:50) Yes, that's it, rebuilding whole of KKFD takes 5 hours if you get lucky, 7 hours if libqt4 needs a rebuild. The winner is kdeutils4. Well, actually it's not so bad anymore, as there were many unnecessary dependencies on kdebase4-workspace-devel and I've already removed them. So kdesdk4 builds directly after kdepimlibs4 and kdeutils4 is down to 6 hours, but still the winner. But the kdevplatform dependency on kdesdk4 was on kompare, and that's been "temporarily" disabled upstream for 3 months. So now kdeutils4 is at 5:30, matching kdepim4 and some other packages have a real dependency on kdebase4- workspace-devel and so they are stuck to be built after 5 hours. Another problem was a long KDE:Qt rebuild, and the aggregate in KKFD seems to stop being blocked only after all of KDE:Qt has been rebuilt (even packages that are actually not aggregated, for whatever reason). The most interesting problem in KDE:Qt is that the Qt libraries themselves, including WebKit, get rebuilt in every libqt4* package. Since things like qt- declarative had a dependency on libqt4-devel-doc, they needed to wait for double Qt rebuild. However, most of the libqt4-devel-doc dependencies were unnecessary, so I have removed them. Actually, the table above is not correct for the very initial state, this is after this fix, it originally took even longer. The table is for the current state, as qt-creator still has a build requirement on libqt4-devel-doc because of help generation tools. Now, that's all that has been already fixed in the repos. But I have more improvements under home:llunak:branches that I haven't committed to KDE:Qt/KKFD yet, as I'd like to get those okayed first. Changes in KDE:Qt should be moving the help generation tools from libqt4- devel-doc to libqt4-devel, where they IMO belong anyway, if apps need them for building documentation during their build. Second change should be making sure that Qt libs and WebKit are built only once. The trick is to install libqt4- devel (and libQtWebKit-devel) and symlinking them to the build dir's bin/ and lib/. It seems to work just fine. I additionally split off libqt4-webkit, in order to have libqt4-devel ready sooner for some further packages that don't require WebKit, although I'm not entirely sure it's worth it (most stuff requires WebKit anyway, and the aggregate blocks KKFD anyway). Changes in KKFD should be using final builds (i.e. KDE4_ENABLE_FINAL=1) for kdelibs4, kdepimlibs4 and kdebase4-workspace. These 3 are the biggest bottlenecks (maybe with python-kde4, but I think kdebindings4 is already troublesome enough). It needed some fixes (it seems nobody has been using that lately, maybe cmake's stupid dependency handling is to blame), but now it builds fine. Hopefully it's not a big deal to keep it that way (it's mostly just kde4_no_enable_final(<whatever>) in the subdir's CMakeLists.txt or just disabling it altogether for the package). Do you think it'd be worth it? We used to build KDE3 this way too. With all these changes, the final table should look like this: libqt4 0:30 (+0:30) -> libqt4-webkit 1:00 (+0:30) -> libqt4-devel-doc 1:20 (+0:20) -> automoc4, soprano, strigi, phonon 1:40 (+0:20) -> kdelibs4 2:00 (+0:20) -> kdepimlibs4 2:10 (+0:10) -> kdebase4-workspace 2:30 (+0:20) -> kdevplatform 2:30 (+0:20) -> python-kde4 3:15 (+0:45) -> kdeutils4 3:35 (+0:20) -> kdepim4-runtime 2:30 (+0:20) -> kdepim4 3:40 (+1:10) -> kdegraphics4 2:25 (+0:25) -> koffice2 4:15 (+1:50) The winner is now koffice2 with 4:15 hours, with no obvious dependency to improve, but it doesn't get updated together with KDE SC, so koffice2 doesn't matter that much. The rest should be under 4 hours total, 2:30 in KKFD alone. Comments, okays, cookies :) ? -- Lubos Lunak KDE developer -------------------------------------------------------------- SUSE LINUX, s.r.o. e-mail: l.lunak@suse.cz , l.lunak@kde.org Lihovarska 1060/12 tel: +420 284 028 972 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kde+help@opensuse.org
On Friday 25 December 2009, Lubos Lunak wrote:
Another problem was a long KDE:Qt rebuild, and the aggregate in KKFD seems to stop being blocked only after all of KDE:Qt has been rebuilt (even packages that are actually not aggregated, for whatever reason).
Yes, thats a strange limitation of the buildservice. as the aggregate is not actually necessary (as it also builds against the project, and this method doesn't block) for correct building, it is only there to make the repo self- contained (so that not KDE:Qt and KKFD have to be added for older repos). The usual fix when you don't want to wait is: disable the aggregate temporarily. Therefore I don't consider optimisations of Qt project (!) rebuild time as extremely important.
The most interesting problem in KDE:Qt is that the Qt libraries themselves, including WebKit, get rebuilt in every libqt4* package.
yep.
Since things like qt- declarative had a dependency on libqt4-devel-doc, they needed to wait for double Qt rebuild.
Thats bad indeed.
However, most of the libqt4-devel-doc dependencies were unnecessary, so I have removed them.
which? I only see this change: Name: libqt4-devel-doc BuildRequires: cups-devel freeglut-devel libjpeg-devel libpng-devel -BuildRequires: libqt4-devel alsa-devel gtk2-devel sqlite3-devel +BuildRequires: libqt4-devel alsa-devel gtk2-devel sqlite3-devel libQtWebKit- devel %if 0%{?suse_version} > 1020
Changes in KDE:Qt should be moving the help generation tools from libqt4- devel-doc to libqt4-devel, where they IMO belong anyway, if apps need them for building documentation during their build.
I was told that they're also needed during runtime. I don't want to pull in libqt4-devel for normal runtime usage. If that is incorrect, then the move would be a good idea indeed. I guess some grepping of source code is in order to verify that the tools are not called during runtime.
Second change should be making sure that Qt libs and WebKit are built only once. The trick is to install libqt4- devel (and libQtWebKit-devel) and symlinking them to the build dir's bin/ and lib/. It seems to work just fine. I additionally split off libqt4-webkit, in order to have libqt4-devel ready sooner for some further packages that don't require WebKit, although I'm not entirely sure it's worth it (most stuff requires WebKit anyway, and the aggregate blocks KKFD anyway).
I don't think splitting off webkit saves significant time, thats why we changed this for 4.6, it used to be built seperately. building of Qt is complex enough, the fewer spec files the better.
(it's mostly just kde4_no_enable_final(<whatever>) in the subdir's CMakeLists.txt or just disabling it altogether for the package). Do you think it'd be worth it? We used to build KDE3 this way too.
yeah, and we always had to fix it (wrong #defines, static's that globber each other and other things introducing weird bugs), and of course, upstream always blamed our packages rather than the actual reason for the bug. I'd rather not use it unless it has a *significant* effect on either runtime performance or compile time.
With all these changes, the final table should look like this: libqt4 0:30 (+0:30) -> libqt4-webkit 1:00 (+0:30) -> libqt4-devel-doc 1:20 (+0:20)
so 30 minute improvement here. I think thats significant. I'm fine with adding your symlink trick to the libqt4-devel-doc.spec.in. in general, all these "black magic tricks" appears to be fragile, as it usually breaks in some way with the next Qt update. I had several of them, and got rid of them one after the other again. This is usually wasting me a lot of time (in the range of hours or days), which I consider worse for my productivity then having to wait 30 minutes for a build to finish (during which I can do something else, like enjoying frozen bubble ;) ). Greetings, Dirk -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kde+help@opensuse.org
On Saturday 02 of January 2010, Dirk Müller wrote:
On Friday 25 December 2009, Lubos Lunak wrote:
Another problem was a long KDE:Qt rebuild, and the aggregate in KKFD seems to stop being blocked only after all of KDE:Qt has been rebuilt (even packages that are actually not aggregated, for whatever reason).
The usual fix when you don't want to wait is: disable the aggregate temporarily. Therefore I don't consider optimisations of Qt project (!) rebuild time as extremely important.
I see, I didn't know that. That would have made this a lot simpler.
The most interesting problem in KDE:Qt is that the Qt libraries themselves, including WebKit, get rebuilt in every libqt4* package.
yep.
Since things like qt- declarative had a dependency on libqt4-devel-doc, they needed to wait for double Qt rebuild.
Thats bad indeed.
However, most of the libqt4-devel-doc dependencies were unnecessary, so I have removed them.
which? I only see this change:
I meant other packages in KDE:Qt depending on libqt4-devel-doc.
Changes in KDE:Qt should be moving the help generation tools from libqt4-devel-doc to libqt4-devel, where they IMO belong anyway, if apps need them for building documentation during their build.
I was told that they're also needed during runtime. I don't want to pull in libqt4-devel for normal runtime usage. If that is incorrect, then the move would be a good idea indeed. I guess some grepping of source code is in order to verify that the tools are not called during runtime.
I don't see any relevant reference to the tools from the source itself, only docs and makefiles. So I think it's ok to move them (and even if not, they could be made into a libqt4-devel subpackage that both the packages would depend on).
Second change should be making sure that Qt libs and WebKit are built only once. The trick is to install libqt4- devel (and libQtWebKit-devel) and symlinking them to the build dir's bin/ and lib/. It seems to work just fine. I additionally split off libqt4-webkit, in order to have libqt4-devel ready sooner for some further packages that don't require WebKit, although I'm not entirely sure it's worth it (most stuff requires WebKit anyway, and the aggregate blocks KKFD anyway).
I don't think splitting off webkit saves significant time, thats why we changed this for 4.6, it used to be built seperately. building of Qt is complex enough, the fewer spec files the better.
Ok, I won't add it then.
(it's mostly just kde4_no_enable_final(<whatever>) in the subdir's CMakeLists.txt or just disabling it altogether for the package). Do you think it'd be worth it? We used to build KDE3 this way too.
yeah, and we always had to fix it (wrong #defines, static's that globber each other and other things introducing weird bugs), and of course, upstream always blamed our packages rather than the actual reason for the bug.
I occassionally used it with KDE3 and I don't remember it breaking often. And now it was only like 10 breakages, and that was probably after nobody caring about it for 2 years (since cmake's dependency handling makes final mode really suck for development). And I wasn't suggesting to build everything that way, just the 3 basic packages.
I'd rather not use it unless it has a *significant* effect on either runtime performance or compile time.
It reduces kdelibs4 from 1:20 to 0:20 and kdebase4-workspace from 1:00 to 0:20 (and that's including the buildroot setup, a local rebuild with icecream could probably do it in 5 minutes if I fixed the meinproc OOM problem). But ok, I won't enable it then, but I think I will still add it to the .spec at least with a %define.
With all these changes, the final table should look like this: libqt4 0:30 (+0:30) -> libqt4-webkit 1:00 (+0:30) -> libqt4-devel-doc 1:20 (+0:20)
so 30 minute improvement here. I think thats significant. I'm fine with adding your symlink trick to the libqt4-devel-doc.spec.in.
in general, all these "black magic tricks" appears to be fragile, as it usually breaks in some way with the next Qt update. I had several of them, and got rid of them one after the other again.
I don't consider it fragile. The Qt symlinking should be perfectly fine, there are even Makefile targets for just rebuilding parts of Qt without any of their dependencies. Do you want the changes also somewhere else other than KDE:Qt?
This is usually wasting me a lot of time (in the range of hours or days), which I consider worse for my productivity then having to wait 30 minutes for a build to finish (during which I can do something else, like enjoying frozen bubble ;) ).
-- Lubos Lunak openSUSE Boosters team, KDE developer l.lunak@suse.cz , l.lunak@kde.org -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kde+help@opensuse.org
On Wednesday 06 January 2010, Lubos Lunak wrote:
I don't see any relevant reference to the tools from the source itself, only docs and makefiles. So I think it's ok to move them (and even if not, they could be made into a libqt4-devel subpackage that both the packages would depend on).
yes, thats true.
I occassionally used it with KDE3 and I don't remember it breaking often. And now it was only like 10 breakages, and that was probably after nobody caring about it for 2 years (since cmake's dependency handling makes final mode really suck for development). And I wasn't suggesting to build everything that way, just the 3 basic packages.
Oh, I didn't that. I think it is fine to use it for everything up to kdebas4- workspace, to optimize rebuild time. Greetings, Dirk -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kde+help@opensuse.org
participants (2)
-
Dirk Müller -
Lubos Lunak