[opensuse-factory] Factory install report
Hello, I am planning to run regular (e.g. daily to weekly) semi-automatic tests of current factory .isos to increase installability/testability of those and milestone releases for testers, thus increasing overall quality. Todays score is poor. Out of 5 tested .isos only the i586-NET got past the install stage and even then hung during the printer configuration phase, so 0.5/5 would be a fair total. I run this on KVM with a self-made perl script generating keyboard-input and capturing pictures (currently 2/sec). As a nice side-effect, there can now be seen compact ogg-theora-videos of the current boot/install having 6x speed at http://www3.zq1.de/bernhard/mirror/opensuse/video/ You might also notice the predicted remaining time on the NET-install to jump up and down. Also the "Loading Installation System" part of the NET-boot is suspiciously long at 100%... and does not have a size given. If you are interested in those scripts, you can get them from http://www3.zq1.de/bernhard/git/autoinst/ but they are nowhere near perfect, yet. have a lot of fun -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Mon, Apr 26, 2010 at 01:46:47PM +0200, Bernhard M. Wiedemann wrote:
Hello,
I am planning to run regular (e.g. daily to weekly) semi-automatic tests of current factory .isos to increase installability/testability of those and milestone releases for testers, thus increasing overall quality.
Todays score is poor. Out of 5 tested .isos only the i586-NET got past the install stage and even then hung during the printer configuration phase, so 0.5/5 would be a fair total.
I run this on KVM with a self-made perl script generating keyboard-input and capturing pictures (currently 2/sec). As a nice side-effect, there can now be seen compact ogg-theora-videos of the current boot/install having 6x speed at http://www3.zq1.de/bernhard/mirror/opensuse/video/
today were tested two new NET install .isos (0571,0572) the x86_64 isos still failed early with a yast2/installer segfault the NET-i586-Build0572 now already failed at kexec on another try it did not find hard disks... or might this be a problem with KVM(version 85)? On another system with KVM, the same iso hung before grub menu. that can't be right. So I need to take a deeper look into that, too. On third retry it went past the first reboot and had a black screen - possibly on second stage config like https://bugzilla.novell.com/show_bug.cgi?id=585432 . http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build057... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build057... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build057... feedback welcome -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag 27 April 2010 schrieb Bernhard M. Wiedemann:
today were tested two new NET install .isos (0571,0572) the x86_64 isos still failed early with a yast2/installer segfault the NET-i586-Build0572 now already failed at kexec
Hi, There is no point in testing the NET isos daily if you do not change the download URL. You will test M5 every day. The NET iso only contains the trigger to download factory-snapshot. If you boot the ISO, I think F5 changes the download url and you need to change it to factory to test factory. BTW: I found the issue with the live cds and I hope the next round will boot again. Gretings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 27. April 2010 schrieb Stephan Kulow:
Am Dienstag 27 April 2010 schrieb Bernhard M. Wiedemann:
today were tested two new NET install .isos (0571,0572) the x86_64 isos still failed early with a yast2/installer segfault the NET-i586-Build0572 now already failed at kexec
Hi,
There is no point in testing the NET isos daily if you do not change the download URL. You will test M5 every day. The NET iso only contains the trigger to download factory-snapshot. If you boot the ISO, I think F5 changes the download url and you need to change it to factory to test factory.
Sometimes there isn't a chance to change the download URL because the menu is distorted (BNC 599478) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Tue, Apr 27, 2010 at 03:17:51PM +0200, Stephan Kulow wrote:
Am Dienstag 27 April 2010 schrieb Bernhard M. Wiedemann:
today were tested two new NET install .isos (0571,0572) the x86_64 isos still failed early with a yast2/installer segfault the NET-i586-Build0572 now already failed at kexec
Hi,
There is no point in testing the NET isos daily if you do not change the download URL. You will test M5 every day. The NET iso only contains the trigger to download factory-snapshot. If you boot the ISO, I think F5 changes the download url and you need to change it to factory to test factory.
BTW: I found the issue with the live cds and I hope the next round will boot again.
Gretings, Stephan
Good thing, you reminded me. I had code to change the address to factory, but forgot to include it into these scripts. Thus it is only natural that I hit the bugs from MS5. Todays report includes one completely successful install of openSUSE-NET-x86_64-Build0575 (now with factory) http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Build0... and I had several later NET-installs (both i586 and x86_64) that got pretty far, so might just be some network issue there. It reproducibly stops after getting http://ftp5.gwdg.de/pub/opensuse/factory/repo/oss/suse/i586/OpenOffice_org-c... (seen in squid/access.log) Maybe it only happens with a proxy? Or only on that mirror? Will need more research. With the upcoming fixes on LiveCDs, this should give a pretty good overall. Then there were some temporary problems captured at http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Build0... possibly because of the factory mirror syncing during my test. I remember the debian mirror script http://www.debian.org/mirror/anonftpsync to first sync the package files (/pool) and only in a second step to update everything, including meta-data with --delete-after. The approach on http://en.opensuse.org/Mirror_Setup_Howto appears to only have one step... so it could probably be improved in this regard. However, one annoying thing remains: that downloaded packages are not cached on my proxy (specified on boot-prompt). The reason lies in the typical HTTP-requests used during installation which include a "no-cache" part: GET /pub/opensuse/factory/repo/oss/suse/i586/libgio-2_0-0-2.24.0-1.4.i586.rpm HTTP/1.1 User-Agent: ZYpp 7.0.0 (aria2 version 1.9.1) Accept: */*,application/metalink4+xml,application/metalink+xml Host: ftp5.gwdg.de Cache-Control: no-cache X-ZYpp-AnonymousId: X-ZYpp-DistributionFlavor: Pragma: There is probably some zypper/aria2 config to change that behaviour but I could not find anything in /etc/zypp/zypp.conf . And googling was not helpful either. At least aria2c --http-no-cache=false should help... but where could I add that option? http://en.opensuse.org/Libzypp/Failover tells that ZYPP_ARIA2C=0 could be used, but I am not sure how to inject this environment variable into the grep HIT /var/log/squid/access.log showed factory/repo/oss/boot/i386/ (common|root|cracklib-dict-full.rpm|bind|branding) to be the only files over 1MB size served from cache. Then I found, that squid does have a "reload-into-ims" (and a harder "ignore-reload") option. applied like this: refresh_pattern \.rpm$ 99999999 100% 99999999 reload-into-ims (added above the shipped refresh_pattern lines) This caused more files to be used from cache (after checking server for freshness). Particularly large files (above 1MB) still do get a TCP_MISS - probably because they are requested in chunks by aria2. -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch 28 April 2010 schrieb Bernhard M. Wiedemann:
and I had several later NET-installs (both i586 and x86_64) that got pretty far, so might just be some network issue there. It reproducibly stops after getting http://ftp5.gwdg.de/pub/opensuse/factory/repo/oss/suse/i586/OpenOffice_org- calc-extensions-3.2.0.99.3-1.1.i586.rpm (seen in squid/access.log) Maybe it only happens with a proxy? Or only on that mirror? Will need more research.
Hmm, https://bugzilla.novell.com/show_bug.cgi?id=600502 reports it as it happening also with DVD. I wonder how this can be i586 specific ;( Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Wed, Apr 28, 2010 at 04:41:52PM +0200, Stephan Kulow wrote:
Am Mittwoch 28 April 2010 schrieb Bernhard M. Wiedemann:
and I had several later NET-installs (both i586 and x86_64) that got pretty far, so might just be some network issue there. It reproducibly stops after getting http://ftp5.gwdg.de/pub/opensuse/factory/repo/oss/suse/i586/OpenOffice_org- calc-extensions-3.2.0.99.3-1.1.i586.rpm (seen in squid/access.log) Maybe it only happens with a proxy? Or only on that mirror? Will need more research.
Hmm, https://bugzilla.novell.com/show_bug.cgi?id=600502 reports it as it happening also with DVD. I wonder how this can be i586 specific ;(
Greetings, Stephan
That is the bug. I suppose, fixed packages will be available soon. Added it to http://en.opensuse.org/Bugs:Most_Annoying_Bugs_11.3_dev openSUSE-NET-i586-Build0578 was still failing from it this afternoon. Also in the news: I have managed to tune my autoinst scripts to be able to get both GNOME and KDE LiveCDs beyond the first reboot so that there are now MS6 videos of those: http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-KDE-LiveCD-i686-B... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-GNOME-LiveCD-i686... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-GNOME-LiveCD-x86_... yesterday there was already http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Build0... and network installs are now a lot faster, using virtio instead of e1000 together with the proxy settings and ZYPP_ARIA2C=0 on bootloader args (which saves bandwidth on both sides). current overall installability score is at 5.5/6 -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
overall installability is still at 5.5/6 On Fri, Apr 30, 2010 at 05:52:48PM +0200, Bernhard M. Wiedemann wrote:
On Wed, Apr 28, 2010 at 04:41:52PM +0200, Stephan Kulow wrote:
Am Mittwoch 28 April 2010 schrieb Bernhard M. Wiedemann:
and I had several later NET-installs (both i586 and x86_64) that got pretty far, so might just be some network issue there. It reproducibly stops after getting http://ftp5.gwdg.de/pub/opensuse/factory/repo/oss/suse/i586/OpenOffice_org- calc-extensions-3.2.0.99.3-1.1.i586.rpm (seen in squid/access.log) Maybe it only happens with a proxy? Or only on that mirror? Will need more research.
Hmm, https://bugzilla.novell.com/show_bug.cgi?id=600502 reports it as it happening also with DVD. I wonder how this can be i586 specific ;(
Greetings, Stephan
That is the bug. I suppose, fixed packages will be available soon. Added it to http://en.opensuse.org/Bugs:Most_Annoying_Bugs_11.3_dev
openSUSE-NET-i586-Build0578 was still failing from it this afternoon. openSUSE-NET-i586-Build0586 still failing from it this morning with calc-extensions-3.2.0.99.3-1.2.i586
while https://bugzilla.novell.com/show_bug.cgi?id=595550 is marked as FIXED for 6 days now, Petr Mladek wrote: I have created submit request against Factory on Friday. It has not been accepted yet => it is possible that there is still broken package in Factory. --- Might even be worth, exercising the update-mechanisms with more than test-updates, because the number of dups in bugzilla still went up. Dups might also be caused by bugzilla excluding RESOLVED issues in its default search. Which means extra work for reporters and maintainers. -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch 05 Mai 2010 schrieb Bernhard M. Wiedemann:
while https://bugzilla.novell.com/show_bug.cgi?id=595550 is marked as FIXED for 6 days now, Petr Mladek wrote: I have created submit request against Factory on Friday. It has not been accepted yet => it is possible that there is still broken package in Factory.
This is about request 39227 and was accepted on monday. The next FTP tree will be synced out soonish I hope. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Wed, May 05, 2010 at 09:54:49AM +0200, Stephan Kulow wrote:
Am Mittwoch 05 Mai 2010 schrieb Bernhard M. Wiedemann:
while https://bugzilla.novell.com/show_bug.cgi?id=595550 is marked as FIXED for 6 days now, Petr Mladek wrote: I have created submit request against Factory on Friday. It has not been accepted yet => it is possible that there is still broken package in Factory.
This is about request 39227 and was accepted on monday. The next FTP tree will be synced out soonish I hope.
Greetings, Stephan
The update made its way through on May 6th or so and produced the first 6/6 installability score. Congrats! I further optimized my code to recognize at which stage it is. moved script repo to the more standard name http://www3.zq1.de/bernhard/git/autoinst.git/ Then I reactivated my old partitioning code to make a more complicated, but still rather regular setup with 4 disks having 3 partitions each. /boot is a RAID1 of sd[abcd]1 / is a RAID6 of big sd[abcd]2 (or optionally RAID10 which still fails with 11.2) swap is a RAID0 of sd[abcd]3 This worked fine on KDE-LiveCD and (after a few tries) also on NET-inst http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-KDE-LiveCD-x86_64... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Build0... Unluckily, this puts some extra IO load on the disks, so I will only run the RAID tests once in a while. Then I added next to the videos the autoinst log which is more suitable for further automatic analysis. It shows for example, which keys were pressed and how long it took to get from one screen to another (via the timestamps in the screenshot names). Example for the above RAID6 test: http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-KDE-LiveCD-x86_64... It shows the partitioning to take 221 seconds while the following configuration parts together took just 32 seconds and everything before partitioning made up for another 53 seconds. partially unrelated: on the KDE-LiveCD, boot/i386/loader/isolinux.msg mentions "openSUSE-11.3-livecd-kde" but the target really is named "openSUSE_Live_(KDE)". Furthermore, the "linux" target is not mentioned at all. GNOME-LiveCD is also affected. When I made my bootable SuSE-7.3 CDs seven years ago, I had a cfg-line "F6 isolinux.cfg" to print the config verbatim, so that I had a way to find out those magic strings even when I forgot to update the help text. It is admittedly not very user-friendly, but having an outdated help text without such a tweak is certainly even less user-friendly. -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Hi Bernhard, On Tue, 11 May 2010 12:12:55 +0200 "Bernhard M. Wiedemann" <susefactory@lsmod.de> wrote:
I further optimized my code to recognize at which stage it is. moved script repo to the more standard name http://www3.zq1.de/bernhard/git/autoinst.git/
just for the record: This is pretty cool stuff you are doing there ;) Have fun, seife -- Stefan Seyfried "Any ideas, John?" "Well, surrounding them's out." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Stefan Seyfried ha scritto:
Hi Bernhard,
On Tue, 11 May 2010 12:12:55 +0200 "Bernhard M. Wiedemann" <susefactory@lsmod.de> wrote:
I further optimized my code to recognize at which stage it is. moved script repo to the more standard name http://www3.zq1.de/bernhard/git/autoinst.git/
just for the record: This is pretty cool stuff you are doing there ;)
Have fun,
seife
Hi both, I agree with Stefan: Bernard your Factory testing is really fantastic and I believe Novell should at least send you a special acknowledgement. Cheers, -- Marco Calistri <amdturion> Minds, like bodies, will often fall into a pimpled, ill-conditioned state from mere excess of comfort. -- Charles Dickens -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
yet another installability report yesterdays overall score: 5.7/6 todays overall score: 3.7/6 openSUSE-GNOME-LiveCD-i686-Build0602-Media.iso and openSUSE-GNOME-LiveCD-i686-Build0603-Media.iso failed to boot into installer ("failed to detect CD") on 5 of 6 tries, but liveCD worked afterwards http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-GNOME-LiveCD-i686... openSUSE-GNOME-LiveCD-x86_64-Build0603-Media.iso was still working. I also could reproduce this on 1/3 tries on openSUSE-GNOME-LiveCD-i686-Build0577 Was also able to reproduce this on physical hardware (Phenom X4 with USB-stick) by selecting install and giving nohz=off maxcpus=1 at 800x600 (adding mem=700M or mem=1024M appeared to increased probability of failure) Then I saw this morning "YaST was unable to find the correct control file. We are using a fallback one. This should not happen and is worth reporting a bug." Even when I used download.opensuse.org as source. http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build060... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build060... http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Build0... This is probably just temporary. I dropped the -smp 4 parameter to kvm, increasing overall reliability and reproducibility a lot. Nearly any install works now or at least fails the same way on every try. added some application testing. this includes a test for MS6-bug https://bugzilla.novell.com/show_bug.cgi?id=600576 (yast2 lan) which shows that the fix still has not propagated to Factory isos since 2010-04-30 http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-KDE-LiveCD-i686-B... Testing code also now exercises reboot and shutdown. added more data to autoinst log: statuser and statsystem, giving the seconds spent by KVM in user-mode or system-mode. While some of this load is contributed by the screenshots taken during my tests and values vary by about 20 between identical runs, it should allow to notice major performance regressions (such as could be caused by some busy looping background process). It also allows for comparison between different methods: openSUSE-GNOME-LiveCD-i686-Build0600: statuser=405.34 statsystem=22.9 openSUSE-GNOME-LiveCD-x86_64-Build0603: statuser=355.43 statsystem=23.64 openSUSE-KDE-LiveCD-i686-Build0603: statuser=325.76 statsystem=18.7 openSUSE-KDE-LiveCD-x86_64-Build0603: statuser=331.39 statsystem=24.32 openSUSE-NET-i586-Build0604: statuser=1011.36 statsystem=58.58 openSUSE-NET-x86_64-Build0604: statuser=865.95 statsystem=135.92 Showing that NET-installs not only take longer because of the network, but also because they need nearly three times as much CPU time - probably from uncompressing RPMs instead of installing images. Ciao Bernhard M. Wiedemann -- software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Sonntag 16 Mai 2010 schrieb Bernhard M. Wiedemann:
Then I saw this morning "YaST was unable to find the correct control file. We are using a fallback one. This should not happen and is worth reporting a bug." Even when I used download.opensuse.org as source. http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build0 607b.ogv http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build0 607c.ogv http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Buil d0607b.ogv This is probably just temporary.
Using a direct mirror (gwdg) is not a good idea as there is no guarentee that the meta data you get fits to the rpms. On download.o.o we take care of that, but the mirrors sync as they get the data. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Mon, May 17, 2010 at 02:39:21PM +0200, Stephan Kulow wrote:
Am Sonntag 16 Mai 2010 schrieb Bernhard M. Wiedemann:
Then I saw this morning "YaST was unable to find the correct control file. We are using a fallback one. This should not happen and is worth reporting a bug." Even when I used download.opensuse.org as source. http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build0 607b.ogv http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-i586-Build0 607c.ogv http://www3.zq1.de/bernhard/mirror/opensuse/video/openSUSE-NET-x86_64-Buil d0607b.ogv This is probably just temporary.
Using a direct mirror (gwdg) is not a good idea as there is no guarentee that the meta data you get fits to the rpms. On download.o.o we take care of that, but the mirrors sync as they get the data.
Greetings, Stephan
This is why I _did_ try download.o.o to find that it was broken there, too. See second link from above: http://www3.zq1.de/opensuse/video/openSUSE-NET-i586-Build0607c.ogv It is back working, now, though. The direct mirror is mostly to make my caching proxy more effective... but this would not be necessary, if download.o.o redirected to the same (closest) mirror most of the time. However, this appears not to be the case with >5 different mirror hostnames showing up in my squid log. While this is a good thing to spread load for the general public, it has disadvantages in my (caching) case. -- Bernhard M. Wiedemann software engineer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On tiistai, 27. huhtikuuta 2010 16:10:43 Bernhard M. Wiedemann wrote: '>
today were tested two new NET install .isos (0571,0572) the x86_64 isos still failed early with a yast2/installer segfault the NET-i586-Build0572 now already failed at kexec on another try it did not find hard disks... or might this be a problem with KVM(version 85)? On another system with KVM, the same iso hung before grub menu. that can't be right. So I need to take a deeper look into that, too. On third retry it went past the first reboot and had a black screen - possibly on second stage config like https://bugzilla.novell.com/show_bug.cgi?id=585432 .
I have better luck. In morning I installed openSUSE-NET-x86_64-Build0571 normal way into hard disk and into VirtualBox. Both without any problems. Now I installed openSUSE-NET-x86_64-Build0572 into VirtualBox with no problems. Install source download.opensuse.org/factory/repo/oss/ -- Viljo -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
participants (6)
-
Bernhard M. Wiedemann -
Marco Calistri -
Markus Koßmann -
Stefan Seyfried -
Stephan Kulow -
Viljo Mustonen