[opensuse-factory] openSUSE Tumbleweed - Status information
Dear Tumbleweed users and hackers, The latest snapshot, 20160308, has been fully built by OBS and tested by openQA. It has not been released to the users, due to an identified issue: ! ! ! i586 systems fail to boot ! ! ! The changes in the snapshot would lead to believe that the kernel update to version 4.4.4 is at fault but unfortunately the problem is deeper nested. In fact, the issue sneaked into the 0305 snapshot, when binutils 2.26 was checked in (this is also not really confirmed, but based on current findings, everything points in this direction). The binutils update did not cause the kernel to be rebuilt, thus there is no problem on any system having updated to binutils 2.26 - as long as they don't build their kernel for an i586 system There are a couple people working on finding the reason for this break and, more importantly, a solution for it. Until that moment, I'm afraid, we won't be able to release any Tumbleweed Snapshot (except if we all agree and accept that i586 is of no importance to Tumbleweed and we can live with this breakage). The obvious 'revert' paths are not as simple as I wish they were: * Reverting to kernel 4.4.3 - won't help, as it will still be rebuilt with binutil 2.26, thus having the same issue * Reverting to binutils 2.25 - I had to do that already once in a staging and it took like 2 weeks to recover - as gcc is built against binutils 2.26 and won't be able to link anything once binutils 2.25 is being used - making bootstrapping a complete pain (it's easier moving forward in versions than backwards). So, until a proper / good fix is found, I ask all the i586-interested people to find the time and donate their resources to actually make this work again... this is THE opportunity to show that the community wants i586 to survive. Cheers, Dominique PS: all findings are documented in bugzilla, https://bugzilla.opensuse.org/show_bug.cgi?id=970239 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Is it in principal possible to uncouple i586 from x86_64 to not block snapshots for the rest of the world? Well, would be interesting to see whether somebody jumps in. When I remember correctly, Mozilla Firefox for i586 did not build for some weeks.... Thanx for your great work, DimStar! -- Jabber: robby81@jabber.de -- Jede meiner Mails ist mittels PGP signiert. Die Signatur kann einfach mit meinem öffentlichen Schlüssel auf den Standard Key-Servern geprüft werden und zum Senden von verschluesselten Mails an mich genutzt werden. -- On Mittwoch, 9. März 2016 15:28:01 CET Dominique Leuenberger / DimStar wrote:
Dear Tumbleweed users and hackers,
The latest snapshot, 20160308, has been fully built by OBS and tested by openQA. It has not been released to the users, due to an identified issue:
! ! ! i586 systems fail to boot ! ! !
The changes in the snapshot would lead to believe that the kernel update to version 4.4.4 is at fault but unfortunately the problem is deeper nested.
In fact, the issue sneaked into the 0305 snapshot, when binutils 2.26 was checked in (this is also not really confirmed, but based on current findings, everything points in this direction).
The binutils update did not cause the kernel to be rebuilt, thus there is no problem on any system having updated to binutils 2.26 - as long as they don't build their kernel for an i586 system
There are a couple people working on finding the reason for this break and, more importantly, a solution for it. Until that moment, I'm afraid, we won't be able to release any Tumbleweed Snapshot (except if we all agree and accept that i586 is of no importance to Tumbleweed and we can live with this breakage).
The obvious 'revert' paths are not as simple as I wish they were: * Reverting to kernel 4.4.3 - won't help, as it will still be rebuilt with binutil 2.26, thus having the same issue * Reverting to binutils 2.25 - I had to do that already once in a staging and it took like 2 weeks to recover - as gcc is built against binutils 2.26 and won't be able to link anything once binutils 2.25 is being used - making bootstrapping a complete pain (it's easier moving forward in versions than backwards).
So, until a proper / good fix is found, I ask all the i586-interested people to find the time and donate their resources to actually make this work again... this is THE opportunity to show that the community wants i586 to survive.
Cheers, Dominique
PS: all findings are documented in bugzilla, https://bugzilla.opensuse.org/show_bug.cgi?id=970239
On Wed, 2016-03-09 at 18:12 +0100, Robby Engelmann wrote:
Is it in principal possible to uncouple i586 from x86_64 to not block snapshots for the rest of the world?
Sadly, not with the current setup: the FTP Tree is built as one product (which is also important as the -32bit packages are actually 'repackaged' i586 packages, splitting this might get us in very strange dep issues)
Well, would be interesting to see whether somebody jumps in. When I remember correctly, Mozilla Firefox for i586 did not build for some weeks....
I considered Firefox much less critical to release the snapshot even with a non-building package; non-building means the 'old' package stays in the repo. A non-booting kernel is pretty nasty - and as it builds, the old kernel is replaced in the repositories. Well, let's see if somebody jumps in - so far I heard of somebody willing to do tests - but nobody preparing anything to test (ok, that's not entirely true - I got a binutils submission to test - should get a kernel build based on that in about 30 minutes) Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 09.03.2016 18:19, Dominique Leuenberger / DimStar wrote:
Well, let's see if somebody jumps in - so far I heard of somebody willing to do tests - but nobody preparing anything to test (ok, that's not entirely true - I got a binutils submission to test - should get a kernel build based on that in about 30 minutes)
I would love to help, but I'm a bit at at loss on what to do best. * install 13.2 * update to Factory but keep old kernel * build SUSE kernel, check if it fails on boot * retry with $PATCH / $ASSEMBLER_OPTION / whatever? Would that be a way to go forward? But then I'd get "latest released" which still works thanks to openQA... How can I get the broken stuff, so that I can test? Best regards, Stefan -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 2016-03-10 at 14:08 +0100, Stefan Seyfried wrote:
I would love to help, but I'm a bit at at loss on what to do best.
* install 13.2 * update to Factory but keep old kernel * build SUSE kernel, check if it fails on boot * retry with $PATCH / $ASSEMBLER_OPTION / whatever?
Steps I did to reproduce locally and work on it: * grab the latest TW -NET installer (0307) (make sure to take the i586) * Install a minimal server / textmode (you don't need more to see it) * add 'osc' to the system * osc getbinaries openSUSE:Factory kernel-default standard i586 * install the new kernel and try to boot it... => it should hang (well, it should not, but it does :P ) * from there on, anything you can do to make it go away that we can put in a package in the end is welcome On my test VM I now have a git kernel tree and the build tools, cloned the openSUSE kernel-default config into it and built. The resulting kernel hangs - so that's a good sign that it is reproducible at least. Hope that helps, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 10/03/16 01:28, Dominique Leuenberger / DimStar wrote:
Dear Tumbleweed users and hackers,
The latest snapshot, 20160308, has been fully built by OBS and tested by openQA. It has not been released to the users, due to an identified issue:
! ! ! i586 systems fail to boot ! ! !
The changes in the snapshot would lead to believe that the kernel update to version 4.4.4 is at fault but unfortunately the problem is deeper nested. [pruned]
Dominique, I don't know if this is relevant in any way to this problem but several days ago kernel-default-4.4.4-1.3 became available (I use the kernel from ../Kernel:/stable/standard ) but the strange part was that zypper would not upgrade all the associated files with this version of the kernel. Let me explain. I use zypper/refresh/patch/up daily to check for new files. On the first day zypper up showed that- kernel-devel-4.4.4-1.3 and kernel-macros would be installed but that- kernel-default-4.4.4-1.3 kernel-default-devel-4.4.4.1-3 would NOT be installed. This occurred over the next 3 days which I found to be very strange because the "missing" files were sitting in the ../stable/standard/ repo. So I then went to YaST2 -- and YaST2 installed all the 4 files without any problems. Why suddenly the big difference in behaviour in zypper and YaST2? -- a rhetorical question :-) . BC -- Using openSUSE 13.2, KDE 4.14.9 & kernel 4.4.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX660 GPU -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 10.03.2016 06:02, Basil Chupin wrote:
Dominique, I don't know if this is relevant in any way to this problem
No, it's not. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 11/03/16 00:09, Stefan Seyfried wrote:
On 10.03.2016 06:02, Basil Chupin wrote:
Dominique, I don't know if this is relevant in any way to this problem No, it's not.
(Finally got to read your response.) OK, thanks. BC -- Using openSUSE 13.2, KDE 4.14.9 & kernel 4.5.0-3 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX660 GPU -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 2016-03-10 at 16:02 +1100, Basil Chupin wrote:
Dominique, I don't know if this is relevant in any way to this problem
I don't think 'related' but 'tangential interessting' might describe it.
but several days ago kernel-default-4.4.4-1.3 became available (I use the kernel from ../Kernel:/stable/standard ) but the strange part was that zypper would not upgrade all the associated files with this version of the kernel. Let me explain.
Unfortunately, you miss one very interesting information for me: is this also the kernel-default i586 package? IF so, then I'm surprised and shocked that this works - the latest version has been built with binutils 2.26 as well; in fact I'd expect kernel:Stable/kernel-default i586 to expose the same issue If of course you run x86_64, then this is clear why this problem does not appear to bite you. As for the update problem you saw: no idea - the provided info would not let me create any assumption what might have been wrong. Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 11/03/16 00:27, Dominique Leuenberger / DimStar wrote:
On Thu, 2016-03-10 at 16:02 +1100, Basil Chupin wrote:
Dominique, I don't know if this is relevant in any way to this problem I don't think 'related' but 'tangential interessting' might describe it.
but several days ago kernel-default-4.4.4-1.3 became available (I use the kernel from ../Kernel:/stable/standard ) but the strange part was that zypper would not upgrade all the associated files with this version of the kernel. Let me explain. Unfortunately, you miss one very interesting information for me: is this also the kernel-default i586 package?
No, I now only have 64-bit machines. While I run only the stock standard openSUSE 13.2 I do use the kernels from ../stable/standard. (BTW, my signature line contains the details about my system but then not everyone is able to see signature lines :-( .)
IF so, then I'm surprised and shocked that this works - the latest version has been built with binutils 2.26 as well; in fact I'd expect kernel:Stable/kernel-default i586 to expose the same issue
If of course you run x86_64, then this is clear why this problem does not appear to bite you.
As for the update problem you saw: no idea - the provided info would not let me create any assumption what might have been wrong.
Yes, "know the feeling". This happened again a day or so again with YaST2, but not zypper, being able to upgrade the kernel with all the 4 files; but no big deal because now that I am aware that this is now a natural behaviour for oS (oS 13.2 at least) I simply use YaST2 to upgrade the kernel.
Cheers, Dominique
OK, already, you are making me break a habit of many years and will therefore now add a salutation to the end of my posts :-) . Ciao, BC -- Using openSUSE 13.2, KDE 4.14.9 & kernel 4.5.0-3 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX660 GPU -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Dominique On 2016-03-09 15:28, Dominique Leuenberger / DimStar wrote:
The latest snapshot, 20160308, has been fully built by OBS and tested by openQA. It has not been released to the users, due to an identified issue:
! ! ! i586 systems fail to boot ! ! !
Is there any news on this issue? Bo -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Bo, and everybody else interested, On Wed, 2016-03-16 at 10:44 +0100, Bo Simonsen wrote:
The latest snapshot, 20160308, has been fully built by OBS and
tested by openQA. It has not been released to the users, due to an identified issue:
! ! ! i586 systems fail to boot ! ! ! Is there any news on this issue?
Progress is slow, much slower than we all wish. What happened so far is that various scenarios could be tested with more or less success. One of the main binutils maintainers took on the case as well and last night I have seen a patch uploaded to the kernel bug tracker. So our next steps will be to validate the patch and convince our kernel team to add this to our package - then we might be fortunate to get Tumbleweed rolling soon again. So, all in all: no change in the situation in Tumbleweed directly, but clearly work in the right direction is done and possible solutions are being found. Cheers, Dominique For reference, the relevant bug reports: https://bugzilla.opensuse.org/show_bug.cgi?id=970239 https://sourceware.org/bugzilla/show_bug.cgi?id=19807 https://bugzilla.kernel.org/show_bug.cgi?id=114671 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Dominique On 2016-03-16 10:59, Dominique Leuenberger / DimStar wrote:
! ! ! i586 systems fail to boot ! ! ! Is there any news on this issue?
One of the main binutils maintainers took on the case as well and last night I have seen a patch uploaded to the kernel bug tracker. So our next steps will be to validate the patch and convince our kernel team to add this to our package - then we might be fortunate to get Tumbleweed rolling soon again.
So, all in all: no change in the situation in Tumbleweed directly, but clearly work in the right direction is done and possible solutions are being found.
Thanks for the update. Looking forward to see Tumbleweed rolling again. Bo -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 2016-03-09 at 15:28 +0100, Dominique Leuenberger / DimStar wrote:
Dear Tumbleweed users and hackers,
The latest snapshot, 20160308, has been fully built by OBS and tested by openQA. It has not been released to the users, due to an identified issue:
! ! ! i586 systems fail to boot ! ! !
Good morning dear openSUSE users and hackers, First, I want to thank you all for the patience you had in the last couple of days while you all eagerly awaited a new Tumbleweed snapshot. But I am sure you understand the motivation and I did not hear (many) voices asking to ignore the case and release with a broken i586 kernel. There have been a couple of people involved in finding a solution to the problem (Thank you guys! You are the heroes!) and a kernel, that is able to avoid the issue, has been submitted and checeked in to openSUSE:Factory yesterday afternoon. The first .iso images have been produced based on this kernel and I could do already some preliminary tests (before openQA has a chance to do so) - and it is promising: the .ISO I tested BOOTS on i586, with kernel version 4.5.0 (yes! The next snapshot will bring you this gem too). So, this is to let you know that I do expect a snapshot (likely 0317 - let's call it St. Patrick) to be released within the next 24 hours. OBS is still building it, as there are quite large changes still added - but the daily snapshots we had on x86_64 promise it not to be in a bad shape. So, get ready - as weekend is update time again. Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (5)
-
Basil Chupin -
Bo Simonsen -
Dominique Leuenberger / DimStar -
Robby Engelmann -
Stefan Seyfried