Hi, here's the problems I ran into. They all seem to be related to systemd, but I did not want to hijack the other thread, although this one might solve the other problem, too.
1. The final kexec/reboot is not executed. Both <final_reboot config:type="boolean">false</final_reboot>, meaning kexec will be used, and <final_reboot config:type="boolean">true</final_reboot> have no effect at all[0]. This also means that...
2. Init-Services (<runlevel>) defined in autoyast are not started/stopped, so, f.e., no ssh keys are generated. (This also results in a lot of subsequent errors in our setup.)
To fix this, just include: <software> <remove-packages config:type="list"> <package>systemd-sysvinit</package> </remove-packages> </software> This way, sysvinit-init will be installed and used.
When can we expect autoyast to support systemd?
[0]: Once you do a manual reboot (if you happen to have access other than ssh to the machine), the machine will do the autoyast kexec/reboot after that.
on Tuesday 22 November 2011 686f6c6d wrote:
When can we expect autoyast to support systemd?
can you try to add a chroot-script and report me if it helps?
Here it helps and then I know how to fix it but systemd seems to be more magic than it looks like on the first sight and I need to be sure it works everywhere.
<chroot-scripts config:type="list"> <script> <chrooted config:type="boolean">true</chrooted> <interpreter>shell</interpreter> <source><![CDATA[ insserv autoyast ]]></source> </script>
On Tue, Nov 22, 2011 at 14:42, Uwe Gansert ug@suse.de wrote:
can you try to add a chroot-script and report me if it helps?
Here it helps and then I know how to fix it but systemd seems to be more magic than it looks like on the first sight and I need to be sure it works everywhere.
<chroot-scripts config:type="list"> <script> <chrooted config:type="boolean">true</chrooted> <interpreter>shell</interpreter> <source><![CDATA[ insserv autoyast ]]></source> </script>
Ah, I misunderstood your reply in the other thread. Removing my <remove-package> entry for systemd and inserting your snippet and, of course, "</chroot-scripts>" changes things, but it does not work completely as it did in 11.4:
When installing with "<final_reboot config:type="boolean">false</final_reboot>" (as we did until now), (1.) the kexec at the end still does not seem to work, so (2.) services are not started (but apart from that, the <runlevel> config seems to be applied, which is an improvement) and so (3.) no ssh keys are generated. After manual reboot I did not see the belated execution of kexec as before. (4.) Creating the ssh keys does not work: The ECDSA ssh key is generated during reboot, but for some reason creating the DSA key fails and so the sshd fails to start (also on subsequent reboots).
When installing with "<final_reboot config:type="boolean">true</final_reboot>", I am seeing Heisenbugs: - The first time, the machine rebooted just fine and everything worked like a charm. - The next few times (after commenting out unrelated stuff in my autoyast profile; yes, I am sure I didn't accidentally break things, I'm using diffs and all those shiny things), I saw the same behavior as for kexec, which is to say it did not work because ssh did not come up, seemingly because of the missing DSA key. - Then, during another install, the machine complained about the missing DSA key again, but the sshd came up anyway. Repeated installations behaved the same way: The error was displayed, but it in the end it worked.
(The sshd error message is, for all the described cases: "Generating /etc/ssh/ssh_host_dsa_key. DSA keys must be 1024 bits Starting SSH daemonCould not load host key: /etc/ssh/ssh_host_dsa_key done" (or, most of the time, "fail").)
When using sysvinit-init now, I see the same ssh error message as with systemd. This wasn't there earlier today with sysvinit, but at least now systemd and sysvinit behave the same way.
on Tuesday 22 November 2011 686f6c6d wrote:
When installing with "<final_reboot config:type="boolean">false</final_reboot>" (as we did until now), (1.) the kexec at the end still does not seem to work,
So do I understand you correctly, that you use final_reboot to fix a broken kexec after the 1st stage? did you try to skip the kexec with forceboot? <forceboot config:type="boolean">true</forceboot>
so (2.) services are not started (but apart from that, the <runlevel> config seems to be applied, which is an improvement) and so (3.) no ssh keys are generated.
I can not reproduce this ssh problem here. Maybe you can send me your complete package list. It's probably a problem with a combination of packages like apparmor or so. This is the package list I am using and I don't see a problem when I use the chroot script:
<software> <do_online_update config:type="boolean">true</do_online_update> <patterns config:type="list"> <pattern>base</pattern> </patterns> <packages config:type="list"> <package>yast2-slp</package> <package>yast2-bootloader</package> <package>yast2-ncurses</package>
<package>less</package> <package>zypper</package> <package>vim</package> </packages> <post-packages config:type="list"> <package>dnsmasq</package> </post-packages> </software>·
After manual reboot I did not see the belated execution of kexec as before. (4.) Creating the ssh keys does not work: The ECDSA ssh key is generated during reboot, but for some reason creating the DSA key fails and so the sshd fails to start (also on subsequent reboots).
at the end of the installation I have:
vbox-ug:~ # l /etc/ssh/ insgesamt 172 drwxr-xr-x 2 root root 4096 23. Nov 09:39 ./ drwxr-xr-x 72 root root 4096 23. Nov 09:39 ../ -rw------- 1 root root 125811 29. Okt 17:40 moduli -rw-r--r-- 1 root root 3056 29. Okt 17:40 ssh_config -rw-r----- 1 root root 3825 29. Okt 17:40 sshd_config -rw------- 1 root root 668 28. Okt 10:25 ssh_host_dsa_key -rw-r--r-- 1 root root 610 28. Okt 10:25 ssh_host_dsa_key.pub -rw------- 1 root root 227 23. Nov 09:39 ssh_host_ecdsa_key -rw-r--r-- 1 root root 174 23. Nov 09:39 ssh_host_ecdsa_key.pub -rw------- 1 root root 985 22. Nov 16:54 ssh_host_key -rw-r--r-- 1 root root 650 22. Nov 16:54 ssh_host_key.pub -rw------- 1 root root 1675 22. Nov 16:54 ssh_host_rsa_key -rw-r--r-- 1 root root 402 22. Nov 16:54 ssh_host_rsa_key.pub
When installing with "<final_reboot config:type="boolean">true</final_reboot>", I am seeing Heisenbugs:
- The first time, the machine rebooted just fine and everything worked
like a charm.
- The next few times (after commenting out unrelated stuff in my
autoyast profile; yes, I am sure I didn't accidentally break things, I'm using diffs and all those shiny things), I saw the same behavior as for kexec, which is to say it did not work because ssh did not come up, seemingly because of the missing DSA key.
- Then, during another install, the machine complained about the
missing DSA key again, but the sshd came up anyway. Repeated installations behaved the same way: The error was displayed, but it in the end it worked.
do you use any special linuxrc parameters? Like doing a VNC or SSH installation?
When using sysvinit-init now, I see the same ssh error message as with systemd. This wasn't there earlier today with sysvinit, but at least now systemd and sysvinit behave the same way.
weird
On Wed, Nov 23, 2011 at 10:54, Uwe Gansert ug@suse.de wrote:
on Tuesday 22 November 2011 686f6c6d wrote:
When installing with "<final_reboot config:type="boolean">false</final_reboot>" (as we did until now), (1.) the kexec at the end still does not seem to work,
So do I understand you correctly, that you use final_reboot to fix a broken kexec after the 1st stage? did you try to skip the kexec with forceboot? <forceboot config:type="boolean">true</forceboot>
Let me try to clarify, you had me confused for a moment there. I hope I get the options right: 1. we never had to reboot after stage 1, so we used the default kexec instead of forceboot there. 2. In pre-12.1 time we used kexec after stage 2 (final_reboot=false), which might be the default anyway, instead of rebooting (final_reboot=true). 3. When my initial tests with 12.1 and systemd didn't work and I noticed that the kexec after stage 2 did not seem to be executed, I started experimenting with final_reboot=true, but the problem seemed to be with the missing "insserv autoyast" instead (both final_reboot stages aren't executed with systemd, unless autoyast is brought up forcefully). If I don't "insserv autoyast", the kexec/reboot is executed after the first manual reboot.
so (2.) services are not started (but apart from that, the <runlevel> config seems to be applied, which is an improvement) and so (3.) no ssh keys are generated.
I can not reproduce this ssh problem here. Maybe you can send me your complete package list. It's probably a problem with a combination of packages like apparmor or so. This is the package list I am using and I don't see a problem when I use the chroot script:
There actually were apparmor errors in the logs (something about "changing hats to unknown" not being allowed, I have no access to the logs at the moment), but after allowing all removed apparmor-* packages to be installed, the problem (and error messages) remained. I will investigate further later on. I'll be away for a few days, but rest assured I'll nag you again in about two weeks. (;
After manual reboot I did not see the belated execution of kexec as before. (4.) Creating the ssh keys does not work: The ECDSA ssh key is generated during reboot, but for some reason creating the DSA key fails and so the sshd fails to start (also on subsequent reboots).
at the end of the installation I have:
vbox-ug:~ # l /etc/ssh/ [...]
I see the same except for the missing DSA keys.
do you use any special linuxrc parameters? Like doing a VNC or SSH installation?
We are installing via VNC. For this particular setup, we have edited the vanilla openSUSE-12.1-DVD-x86_64.iso just to include these options in boot/x86_64/loader/isolinux.cfg, so we have less typing to do during install: ----------- 8< ---------- # install label linux kernel linux append initrd=initrd showopts textmode=1 splash=0 nameserver=[...] gateway=[...] install=http://%5B...%5D/12.1 autoyast=http://%5B...%5D/12.1/ hostip=[...]/24 ---------- >8 ----------
On Thu, Nov 24, 2011 at 09:51, 686f6c6d 686f6c6d@googlemail.com wrote:
- When my initial tests with 12.1 and systemd didn't work and I
noticed that the kexec after stage 2 did not seem to be executed, I started experimenting with final_reboot=true, but the problem seemed to be with the missing "insserv autoyast" instead (both final_reboot stages aren't executed with systemd, unless autoyast is brought up forcefully). If I don't "insserv autoyast", the kexec/reboot is executed after the first manual reboot.
The bottom line until now: - Apparmor wasn't the problem. - "insserv autoyast" improves the reboot behavior very much, but there are still differences in which services are "chkconfig on"-ed and started (but part of the problem is that much of the output has changed and my after-install init-script testing what went wrong can't deal with the new messages yet). - The dsa-key is not an autoyast problem, I can reproduce it during manual install of 12.1, too, with both vanilla -DVD and -net ISOs. (All tests were on qemu, so that might play a role.)
I'll continue systemd/package list testing later, but since we have a working system with sysvinit again, that's low priority and will most likely take a few weeks (working part-time).
On Sun, 11 Dec 2011 23:40:21 +0100 686f6c6d 686f6c6d@googlemail.com wrote:
I'll continue systemd/package list testing later, but since we have a working system with sysvinit again, that's low priority and will most likely take a few weeks (working part-time).
Please let me know if you have an soultion since we are experiencing the same problems with systemd. With your initial tip to remove systemd-init from the install list all works well, we are using this approach for out testing, but with systemd the machine doesn't even start the after-reboot install stage. :( Since stage2 takes care of user setup and other thing we can't even login to the machine after an autoyast setup with systemd.
-- Kind regards.
Conny Seidel
################################################################## # Email : conny.seidel@amd.com GnuPG-Key : 0xA6AB055D # # Fingerprint: 17C4 5DB2 7C4C C1C7 1452 8148 F139 7C09 A6AB 055D # ################################################################## # Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach # # General Managers: Alberto Bozzo # # Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen # # HRB Nr. 43632 # ##################################################################
On Mon, Dec 12, 2011 at 00:06, Conny Seidel conny.seidel@amd.com wrote:
On Sun, 11 Dec 2011 23:40:21 +0100 686f6c6d 686f6c6d@googlemail.com wrote:
I'll continue systemd/package list testing later, but since we have a working system with sysvinit again, that's low priority and will most likely take a few weeks (working part-time).
Please let me know if you have an soultion since we are experiencing the same problems with systemd. With your initial tip to remove systemd-init from the install list all works well, we are using this approach for out testing, but with systemd the machine doesn't even start the after-reboot install stage. :( Since stage2 takes care of user setup and other thing we can't even login to the machine after an autoyast setup with systemd.
Sorry for the *very* late reply, it took me quite longer to get back to this than I hoped. Are you using the "insserv autoyast" chroot-script Uwe recommended? (Message-Id: 201111221442.23566.ug@suse.de)
My own problems are plenty and tedious to debug. This week alone ate 20 work hours:
1. The package "sysconfig" was silently not installed because I had the "vlan" package in remove-packages (and "sysconfig" not explicitly in packages).
2. During install, "kernel-default(-3.1.9-1.4.1)" failed to install. The problem boils down to: ---------- 8< ---------- 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:1045 Downloaded 390/420 packages (92%) 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [Pkg] PackageSlideShow.ycp:1050 Pkg Builtin called: CommitPolicy 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [zypp] PackageProvider.cc(providePackage):170 provided Package (41405)systemd-37-3.11.1.x86_64(http-our-repo-host.example-350e2aad) at /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/systemd-37-3.11.1.x86_64.rpm 2012-03-21 16:19:20 <5> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:1174 pkg_name: systemd 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [Pkg] PackageSlideShow.ycp:1206 Pkg Builtin called: PkgInstalled 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [Pkg] Package.cc(searchPackage):575 Package 'systemd' installed: false 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [zypp] RpmDb.cc(doInstallPackage):1708 RpmDb::installPackage(/var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/systemd-37-3.11.1.x86_64.rpm,0x0000000c) 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(start_program):229 Executing 'rpm' '--root' '/mnt' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--force' '--nodeps' '--' '/var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/systemd-37-3.11.1.x86_64.rpm' 2012-03-21 16:19:20 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(start_program):381 pid 6782 launched 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(checkStatus):482 Pid 6782 successfully completed 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp] PathInfo.cc(unlink):670 unlink /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/systemd-37-3.11.1.x86_64.rpm 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp] PackageProvider.cc(providePackage):119 provide Package (39677)kernel-default-3.1.9-1.4.1.x86_64(http-our-repo-host.example-350e2aad) 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:1268 Package 'kernel-default' is remote 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (39677)kernel-default-3.1.9-1.4.1.x86_64(http-our-repo-host.example-350e2aad) 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp] RepoProvideFile.cc(provideFile):251 [1]./x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm{37.7 MiB|sha256-d99e16cff1e14e74ed4e5cbaabd988c9c60431672f76872cf96dea7ad082fcd9} 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp:fetcher++] Fetcher.cc(addCachePath):319 Adding fetcher cache: '/var/cache/zypp/packages/http-our-repo-host.example-350e2aad'. 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp] RepoProvideFile.cc(provideFile):271 Added cache path /var/cache/zypp/packages/http-our-repo-host.example-350e2aad 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp] RepoProvideFile.cc(provideFile):281 Providing file of repo 'http-our-repo-host.example-350e2aad' from http://our-repo-host.example/suse/update/12.1/ 2012-03-21 16:19:21 <2> 10.11.12.223(3197) [zypp] RepoProvideFile.cc(setVerifierForRepo):215 No media verifier for repo 'http-our-repo-host.example-350e2aad' media/media.1 does not exist in '/var/cache/zypp/raw/http-our-repo-host.example-350e2aad' 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp:fetcher] Fetcher.cc(downloadAndReadIndexList):716 No indexes to read. 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp:fetcher] Fetcher.cc(provideFromCache):350 start fetcher with 1 cache directories. 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp:fetcher] Fetcher.cc(provideToDest):548 Not found in cache, downloading 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaSetAccess.cc(provide):203 Going to try to provide file ./x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm from media number 1 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaManager.cc(checkDesired):112 checkDesired(18): desired (cached) 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaManager.cc(checkDesired):112 checkDesired(18): desired (cached) 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1266 dest: /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1267 temp: /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm.new.zypp.CDM9NU 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaCurl.cc(doGetFileCopyFile):1321 ./x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:21 <1> 10.11.12.223(3197) [zypp++] MediaCurl.cc(doGetFileCopyFile):1331 URL: http://our-repo-host.example/suse/update/12.1/x86_64/kernel-default-3.1.9-1.... 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1302 HTTP response: 200 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] MediaMultiCurl.cc(looks_like_metalink):1227 looks_like_metalink(/mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm.new.zypp.CDM9NU): 0 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp] PathInfo.cc(rename):684 rename /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm.new.zypp.CDM9NU -> /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1418 done: /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm{- 0644 0/0 size 39490210} 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] MediaHandler.cc(provideFile):980 provideFile(./x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm) 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] MediaManager.cc(checkDesired):112 checkDesired(18): desired (cached) 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(start_program):229 Executing '/bin/cp' '--remove-destination' '--' '/mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm' '/var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm' 2012-03-21 16:19:23 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(start_program):381 pid 6792 launched 2012-03-21 16:19:24 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(checkStatus):482 Pid 6792 successfully completed 2012-03-21 16:19:24 <1> 10.11.12.223(3197) [zypp] PathInfo.cc(copy):783 hardlinkCopy /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm -> /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpmcopy /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm -> /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:24 <1> 10.11.12.223(3197) [zypp++] MediaSetAccess.cc(releaseFile):85 Going to release file ./x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm from media number 1 2012-03-21 16:19:24 <1> 10.11.12.223(3197) [zypp] PathInfo.cc(unlink):670 unlink /mnt/var/tmp/AP_0x00000002/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:24 <1> 10.11.12.223(3197) [zypp:fetcher] Fetcher.cc(validate):392 Checking job [/var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm] (1 checkers ) 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [Progress++] ProgressData.cc(report):86 {#491|}END 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [zypp] RepoProvideFile.cc(provideFile):330 provideFile at /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:1045 Downloaded 391/420 packages (93%) 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [Pkg] PackageSlideShow.ycp:1050 Pkg Builtin called: CommitPolicy 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [zypp] PackageProvider.cc(providePackage):170 provided Package (39677)kernel-default-3.1.9-1.4.1.x86_64(http-our-repo-host.example-350e2aad) at /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:25 <5> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:1174 pkg_name: kernel-default 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [Pkg] PackageSlideShow.ycp:1206 Pkg Builtin called: PkgInstalled 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [Pkg] Package.cc(searchPackage):575 Package 'kernel-default' installed: false 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [zypp] RpmDb.cc(doInstallPackage):1708 RpmDb::installPackage(/var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm,0x0000000c) 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(start_program):229 Executing 'rpm' '--root' '/mnt' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--force' '--nodeps' '--' '/var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm' 2012-03-21 16:19:25 <1> 10.11.12.223(3197) [zypp++] ExternalProgram.cc(start_program):381 pid 6793 launched 2012-03-21 16:19:50 <2> 10.11.12.223(3197) [zypp] ExternalProgram.cc(checkStatus):495 Pid 6793 was killed by signal 9 (Killed) 2012-03-21 16:19:51 <5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 RpmDb.cc(doInstallPackage):1825 THROW: Subprocess failed. Error: RPM failed: Free diskspace below /boot: 29494360 blocks 2012-03-21 16:19:51 <5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 2012-03-21 16:19:51 <5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 2012-03-21 16:19:51 <1> 10.11.12.223(3197) [YCP] PackageCallbacks.ycp:537 DonePackage(error: 3, reason: 'Subprocess failed. Error: RPM failed: Free diskspace below /boot: 29494360 blocks
') 2012-03-21 16:19:51 <1> 10.11.12.223(3197) [ui] YPushButton.cc(setRole):172 Guessing function key F9 for YPushButton "Abort" at 0x7fcb900d3610 from button role YCancelButton 2012-03-21 16:19:51 <1> 10.11.12.223(3197) [ui] YPushButton.cc(setRole):172 Guessing function key F10 for YPushButton "Ignore" at 0x7fcb901fd190 from button role YOKButton 2012-03-21 16:19:54 <1> 10.11.12.223(3197) [ui] YPushButton.cc(setRole):172 Guessing function key F9 for YPushButton "Abort" at 0x7fcb900d3750 from button role YCancelButton 2012-03-21 16:19:54 <1> 10.11.12.223(3197) [ui] YPushButton.cc(setRole):172 Guessing function key F10 for YPushButton "Ignore" at 0x7fcb9033b250 from button role YOKButton 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [YCP] PackageCallbacks.ycp:612 DonePackage `ignore 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [ui] YPushButton.cc(setFunctionKey):204 Guessing button role YOKButton for YPushButton "OK" at 0x7fcb90025860 from function key F10 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:932 src #0: [13180928] 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:932 src #1: [] 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:932 src #2: [4066304] 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:932 src #3: [2497536] 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [YCP] PackageSlideShow.ycp:932 src #4: [713728] 2012-03-21 16:19:58 <1> 10.11.12.223(3197) [zypp] PathInfo.cc(unlink):670 unlink /var/cache/zypp/packages/http-our-repo-host.example-350e2aad/x86_64/kernel-default-3.1.9-1.4.1.x86_64.rpm 2012-03-21 16:19:59 <1> 10.11.12.223(3197) [zypp] PackageProvider.cc(providePackage):119 provide Package (41655)yast2-ntp-client-2.21.2-2.3.1.noarch(http-our-repo-host.example-350e2aad) ---------- >8 ----------
How can I debug this further?
3. To find the cause for this error, I a.) ran all my XML files through "yast autoyast", saved and diffed, to find potential syntax changes. There were no significant changes. b.) Next I tried to fiddle with remove-packages and packages, until... c.) ...I finally gave up and commented them all out. Which didn't help either. d.) Manual installation with no package editing /was/ possible, so I guess it either has to do with the packages selection, or an kernel-default rpm bug, or autoyast bug. e.) Changing a few packages results in another, different "Error during initrc creation" or something like that. I didn't investigate further. f.) Right now I'm doing a manual installation, will create a profile from that, and try to merge that into our current profile.
Any hints on how to proceed?
On 22.03.2012 15:53, 686f6c6d wrote:
did you maybe overlook this in the log?
ExternalProgram.cc(checkStatus):495 Pid 6793 was killed by signal 9 (Killed) 2012-03-21 16:19:51<5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 RpmDb.cc(doInstallPackage):1825 THROW: Subprocess failed. Error: RPM failed: Free diskspace below /boot: 29494360 blocks 2012-03-21 16:19:51<5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 2012-03-21 16:19:51<5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 2012-03-21 16:19:51<1> 10.11.12.223(3197) [YCP] PackageCallbacks.ycp:537 DonePackage(error: 3, reason: 'Subprocess failed. Error: RPM failed: Free diskspace below /boot: 29494360 blocks
')
How can I debug this further?
looks like something is wrong with your boot partition. The Kernel was not installed. What's the size of your /boot partition in the XML file?
On Thu, Mar 22, 2012 at 16:31, Uwe Gansert ug@suse.de wrote:
On 22.03.2012 15:53, 686f6c6d wrote:
did you maybe overlook this in the log?
ExternalProgram.cc(checkStatus):495 Pid 6793 was killed by signal 9 (Killed) 2012-03-21 16:19:51<5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 RpmDb.cc(doInstallPackage):1825 THROW: Subprocess failed. Error: RPM failed: Free diskspace below /boot: 29494360 blocks 2012-03-21 16:19:51<5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 2012-03-21 16:19:51<5> 10.11.12.223(3197) [zypp] Exception.cc(log):137 2012-03-21 16:19:51<1> 10.11.12.223(3197) [YCP] PackageCallbacks.ycp:537 DonePackage(error: 3, reason: 'Subprocess failed. Error: RPM failed: Free diskspace below /boot: 29494360 blocks
')
How can I debug this further?
looks like something is wrong with your boot partition. The Kernel was not installed. What's the size of your /boot partition in the XML file?
No, I did see that. I assumed the diskspace info is given because it is reasonable to assume this might be an issue in many cases. But since this VM has a 16gb disk that is partitioned as one big ext3 without problems, I ignored the error message. AFAICT, this is not a disk space or mount point issue. Some subprocess (during kernel %install, I presume) is killed for whatever reason and I don't know why. And I'm unable to find the cause for this behavior and don't know how to get more info. (I assume the kernel is missing a dependency or something like that.) If I select ignore, the installation proceeds fine and on the installed system I cannot reproduce the error. Since this install is supposed to be unattended, I would like very much to hunt that bug down.
On 22.03.2012 16:57, 686f6c6d wrote:
No, I did see that. I assumed the diskspace info is given because it is reasonable to assume this might be an issue in many cases. But since this VM has a 16gb disk that is partitioned as one big ext3
ah, you are right. I did the math wrong and calculated the free blocks to 14MB but it's 14GB That should be enough
without problems, I ignored the error message. AFAICT, this is not a disk space or mount point issue. Some subprocess (during kernel %install, I presume) is killed for whatever reason and I don't know why. And I'm unable to find the cause for this behavior and don't know how to get more info. (I assume the kernel is missing a dependency or something like that.)
if you stop the installation with the confirm dialog, do you see anything suspicious in the proposal screen? Check the "Software" link there too. you can stop the installation by passing "y2confirm" to the linuxrc Maybe you can see some resolver errors or so.
On Thu, Mar 22, 2012 at 17:18, Uwe Gansert ug@suse.de wrote:
On 22.03.2012 16:57, 686f6c6d wrote:
ah, you are right. I did the math wrong and calculated the free blocks to 14MB but it's 14GB That should be enough
(;
if you stop the installation with the confirm dialog, do you see anything suspicious in the proposal screen? Check the "Software" link there too. you can stop the installation by passing "y2confirm" to the linuxrc Maybe you can see some resolver errors or so.
Thanks, I had tried all that before and it showed no problems at all.
Now I ctrl-alt-f2ed when the error occurred, chrooted into /mnt, fetched the RPM manually and tried to install it, and then I finally see some information on screen and in the zypper.log (see attachment). So I guess GRUB or something related to it isn't working properly. Will do more debugging tomorrow.
On Thu, Mar 22, 2012 at 18:26, Hans-Joachim Ehlers HansJoachim.Ehlers@eumetsat.int wrote:
I had on a disk a previous SLES11 installation. I used the disk to install OS 12.1 and failed. I did not dig really deep into it but as i wiped out the disk partition table the following installation was Ok.
Thanks for the tip, but that wasn't it (I had initialize=true anyway, but manually wiping the partitions had no effect, either). It pointed me more in the right direction, though: I recently changed the partition layout (on 11.4), and there were still some inconsistencies in the <partition> and <bootloader> section that *might* be part of the problem (when fiddling with them, GRUB itself threw errors at me). Curiously, on 11.4 these changes did not cause problems, but it seems on 12.1 they do. I'll post my profile diffs here once i get it to work.
Interestingly, my problem with kernel-default really boils down to what the screenshot indicates: There are only a handful of loop* and md* files in /mnt/dev/, but sda1 etc are missing. No clue why or how this happens. Outside of the chroot, the device files exist.
On Fri, Mar 23, 2012 at 15:31, 686f6c6d 686f6c6d@googlemail.com wrote:
Interestingly, my problem with kernel-default really boils down to what the screenshot indicates: There are only a handful of loop* and md* files in /mnt/dev/, but sda1 etc are missing. No clue why or how this happens. Outside of the chroot, the device files exist.
- Using kernel-default-3.1.0-1.2.1.x86_64 from the "install" repo, the installation succeeds. - Using kernel-default-3.1.9-1.4.1.x86_64 from the "update" repo, the installation fails.
(We locally mirror ftp5.gwdg.de, strip out deltarpms because yum doesn't like them and recreate and re-sign repomd.xml etc.; the mirroring itself does not seem to be the problem: c287[...]af7287 kernel-default-3.1.9-1.4.1.x86_64.rpm.d.o.o c287[...]af7287 kernel-default-3.1.9-1.4.1.x86_64.rpm.gwdg c287[...]af7287 kernel-default-3.1.9-1.4.1.x86_64.rpm.internal )
From where I'm standing, it really seems to be one of two options:
1. kernel-default-3.1.9-1.4.1.x86_64 is buggy because it misses a dependency, maybe on a deltarpm. 2. kernel-default-3.1.9-1.4.1.x86_64 is buggy because it contains a logical error, presumably in the specfile.
As a temporary workaround, I'll add the update repo during/after installation and not via <add-on>.
On Mon, Apr 2, 2012 at 16:16, 686f6c6d 686f6c6d@googlemail.com wrote:
- Using kernel-default-3.1.0-1.2.1.x86_64 from the "install" repo, the
installation succeeds.
- Using kernel-default-3.1.9-1.4.1.x86_64 from the "update" repo, the
installation fails.
Let me correct that: Once I changed the profile again to re-include some of our scripts (that were disabled while debugging), even kernel-default 3.1.0 fails with the same problem. Need to do further debugging.
On Tue, Apr 10, 2012 at 1:19 PM, 686f6c6d 686f6c6d@googlemail.com wrote:
On Mon, Apr 2, 2012 at 16:16, 686f6c6d 686f6c6d@googlemail.com wrote:
- Using kernel-default-3.1.0-1.2.1.x86_64 from the "install" repo, the
installation succeeds.
- Using kernel-default-3.1.9-1.4.1.x86_64 from the "update" repo, the
installation fails.
Let me correct that: Once I changed the profile again to re-include some of our scripts (that were disabled while debugging), even kernel-default 3.1.0 fails with the same problem. Need to do further debugging.
When working around the kernel issue by using only the suse "install" repo (and not the "update" repo), I see the following behaviour:
1. /var/log/messages show these errors: "su: pam_systemd(su:session): Failed to create session: Invalid argument" (seems to be https://bugzilla.novell.com/show_bug.cgi?id=731358 )
2. Although "network", "ntp" and "sshd" are shown by chkconfig to be "on" (with ntp and sshd being explicitly specified as "3 5" in <runlevel>), they are not started at boot ("systemctl list-units -t service --all" shows them all as "loaded inactive"). Do I have to change anything else to make yast2-runlevel work with systemd?
3. The network comes up during boot (it works flawlessly in stage2 and /etc/sysconfig/network/ifcfg-eth0 and [...]/routes look fine), but then seems to exit for no particular reason, even when I try to start it manually (with "systemctl start network.service"). The only thing in the logs is this: ---------- 8< ---------- ifup: eth0 ifup: IP address: [...] dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... dns-resolver: You can find my version[...] ifdown: eth0 ---------- >8 ---------- systemctl shows it as "Active: inactive (dead)" then, with "ExecStart=[...], status=0/SUCCESS". I am *so* clueless what happens.
(When I forbid systemd-sysvinit in autoyast, everything works fine as expected again.) I really hope 12.2 will be delayed, otherwise my resume will read "failed to make autoyast work with systemd for 12.1's entire lifetime." (;
On 06/12/2012 01:53 PM, 686f6c6d wrote:
On Tue, Apr 10, 2012 at 1:19 PM, 686f6c6d686f6c6d@googlemail.com wrote:
On Mon, Apr 2, 2012 at 16:16, 686f6c6d686f6c6d@googlemail.com wrote:
- Using kernel-default-3.1.0-1.2.1.x86_64 from the "install" repo, the
installation succeeds.
- Using kernel-default-3.1.9-1.4.1.x86_64 from the "update" repo, the
installation fails.
Let me correct that: Once I changed the profile again to re-include some of our scripts (that were disabled while debugging), even kernel-default 3.1.0 fails with the same problem. Need to do further debugging.
When working around the kernel issue by using only the suse "install" repo (and not the "update" repo), I see the following behaviour:
- /var/log/messages show these errors: "su: pam_systemd(su:session):
Failed to create session: Invalid argument" (seems to be https://bugzilla.novell.com/show_bug.cgi?id=731358 )
- Although "network", "ntp" and "sshd" are shown by chkconfig to be
"on" (with ntp and sshd being explicitly specified as "3 5" in <runlevel>), they are not started at boot ("systemctl list-units -t service --all" shows them all as "loaded inactive"). Do I have to change anything else to make yast2-runlevel work with systemd?
- The network comes up during boot (it works flawlessly in stage2 and
/etc/sysconfig/network/ifcfg-eth0 and [...]/routes look fine), but then seems to exit for no particular reason, even when I try to start it manually (with "systemctl start network.service"). The only thing in the logs is this: ---------- 8< ---------- ifup: eth0 ifup: IP address: [...] dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... dns-resolver: You can find my version[...] ifdown: eth0 ---------->8 ---------- systemctl shows it as "Active: inactive (dead)" then, with "ExecStart=[...], status=0/SUCCESS". I am *so* clueless what happens.
I've seen this a lot, but not all the time, when upgrading systems from 11.3 to 12.1, via "zypper dup". From what I can gather the problem seems to be due to "netconfig update" exiting with a non-zero status due to resolv.conf not matching the NETCONFIG_DNS_POLICY. I've been mostly successful in working around this by setting NETCONFIG_DNS_POLICY="" in /etc/sysconfig/network/config. I say mostly successfully because the "inactive (dead) networking has happened even after setting NETCONFIG_DNS_POLICY="".
This seems to happen more on VM's than on physical servers for me. When dup'ing to 12.1 the VM's don't always like to reboot gracefully so I was doing "reboot -f" after which networking may or may not be "inactive (dead)". I found that when it was dead if I stopped networking and rebooted the VM, networking would function properly subsequently. Also, after dup to 12.1 if I don't "reboot -f" and simply reboot, and once the system hangs up during shutdown if I reset power to the VM networking comes up fine.
Since there's really no verbose|debug logging|output in systemd/systemctl it's REALLY difficult to get to the root of the problem. If anyone has any ideas I'm all ears? I haven't gotten to strace'ing the network.service yet but I have a feeling it may be only way to get an really info about what going on.
(When I forbid systemd-sysvinit in autoyast, everything works fine as expected again.) I really hope 12.2 will be delayed, otherwise my resume will read "failed to make autoyast work with systemd for 12.1's entire lifetime." (;
Thanks for the info! I've been spending hours yesterday finding those systemd bug reports that might apply (damn, there are *many* systemd bugs). I'll have access to the machines for further testing (and to look up the bugzilla IDs) again next monday and will keep you posted if I make any progress.
To follow up on this... (I haven't been able to make our autoyast profile work with the "update" repo enabled yet, so this is with only the suse/install repo, and hence might include bugs that are already fixed in the update repo.)
On 06/12/2012 01:53 PM, 686f6c6d wrote:
- The network comes up during boot (it works flawlessly in stage2 and
/etc/sysconfig/network/ifcfg-eth0 and [...]/routes look fine), but then seems to exit for no particular reason, even when I try to start it manually (with "systemctl start network.service"). The only thing in the logs is this: ---------- 8< ---------- ifup: eth0 ifup: IP address: [...] dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... dns-resolver: You can find my version[...] ifdown: eth0 ---------->8 ---------- systemctl shows it as "Active: inactive (dead)" then, with "ExecStart=[...], status=0/SUCCESS". I am *so* clueless what happens.
The problem with eth0 going down seems to have disappeared on its own. I have no clue why.
- Although "network", "ntp" and "sshd" are shown by chkconfig to be
"on" (with ntp and sshd being explicitly specified as "3 5" in <runlevel>), they are not started at boot ("systemctl list-units -t service --all" shows them all as "loaded inactive"). Do I have to change anything else to make yast2-runlevel work with systemd?
With <final_reboot>, both ntpd and sshd come up fine, so it seems to be a problem at first boot only. Thanks for the tip, darix! (It does not seem to be #724777, #725503, #732930, #735943, #736059, #765039; it might be #727771 or #747931; if not, I'll post a new one.)
On Tue, Jun 12, 2012 at 8:39 PM, Darin Perusich Darin.Perusich@ctg.com wrote:
I've seen this a lot, but not all the time, when upgrading systems from 11.3 to 12.1, via "zypper dup". From what I can gather the problem seems to be due to "netconfig update" exiting with a non-zero status due to resolv.conf not matching the NETCONFIG_DNS_POLICY. I've been mostly successful in working around this by setting NETCONFIG_DNS_POLICY="" in /etc/sysconfig/network/config. I say mostly successfully because the "inactive (dead) networking has happened even after setting NETCONFIG_DNS_POLICY="".
Because our autoyast for 12.1 doesn't work yet, we have actually done a lot of DUPs, but according to our folks in the trenches we never had the problem in that scenario.
This seems to happen more on VM's than on physical servers for me. When dup'ing to 12.1 the VM's don't always like to reboot gracefully so I was doing "reboot -f" after which networking may or may not be "inactive (dead)". I found that when it was dead if I stopped networking and rebooted the VM, networking would function properly subsequently. Also, after dup to 12.1 if I don't "reboot -f" and simply reboot, and once the system hangs up during shutdown if I reset power to the VM networking comes up fine.
We don't have this problem, neither after DUPing nor with fresh 12.1 installs.
Since there's really no verbose|debug logging|output in systemd/systemctl it's REALLY difficult to get to the root of the problem. If anyone has any ideas I'm all ears? I haven't gotten to strace'ing the network.service yet but I have a feeling it may be only way to get an really info about what going on.
A few of all the above bugreports recommend to add "systemd.log_level=debug systemd.log_target=kmsg" to the kernel commandline, but then systemd logs so much that rsyslogd with the default install config can't keep up and drops about 800 lines during boot. It is still helpful though. (Haven't found a bugreport about that yet, will file one in the next weeks if none exists.)
On 20.06.2012 20:49, 686f6c6d wrote:
With <final_reboot>, both ntpd and sshd come up fine, so it seems to be a problem at first boot only. Thanks for the tip, darix! (It does not seem to be #724777, #725503, #732930, #735943, #736059, #765039; it might be #727771 or #747931; if not, I'll post a new one.)
I've created a bugreport for that. https://bugzilla.novell.com/show_bug.cgi?id=769924
autoinstall@lists.opensuse.org
-
686f6c6d -
Conny Seidel -
Darin Perusich -
Hans-Joachim Ehlers -
Uwe Gansert