http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c2 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |milan.zimmermann@gmail.com, | |tiwai@suse.com Flags| |needinfo?(milan.zimmermann@ | |gmail.com) --- Comment #2 from Takashi Iwai --- Could you check whether boot works fine with nomodeset boot option? If yes, the problem is likely radeon kernel driver. Also, (without nomodeset option), try to boot without quiet boot option. This might show the error messages by chance. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c4 --- Comment #4 from Takashi Iwai --- That's surprising. Basically nomodeset skips KMS so essentially the whole graphics driver will be skipped. Your result implies that it's no graphics driver issue. Hmmm... Could you run "hwinfo --all" once when you could boot, and attach the output to Bugzilla? Also, attach the output of "dmesg" after boot, too. Other things to check: does the installation DVD boot properly at every time? What about the rescue system on DVD? Is it only the installed system that hangs? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c6 --- Comment #6 from milan zimmermann --- Created attachment 646716 --> http://bugzilla.opensuse.org/attachment.cgi?id=646716&action=edit hwinfo --al for bug_944659 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c8 --- Comment #8 from milan zimmermann --- Thanks for the info about pressing the 'e' in GRUB. I knew there was a way but did not remember. I am adding two attachements with resuld of hwinfo -- all > hwinfo_bug_944659 dmesg > dmesg_bug_944659 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c10 --- Comment #10 from milan zimmermann --- I also have an update here: Let me look into two things: 1. When I said that I changed DEFAULT_APPEND in sysconfig->bootloader and removed the "quiet" option - I did that, it is still gone. But it appears this did not actually affect the bootloader. The reason I think that is when you pointed out the 'e' option, I tried that removed "quiet", and the system did boot (one try only so this is not conclusive) but most importantly, the boot process looked different, it does show all the messages (which removing "quiet" from the DEFAULT_APPEND). Is there a way for me to remove the "quiet" permanently so I can test more? 2. I noticed my SECURE_BOOT is set to "yes" - I plan to experiment with setting it to "no". I will not have time to experiment more until my evening, but you should get more info by tomorrow. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c12 --- Comment #12 from milan zimmermann --- Ok, I have a few things to follow up on (USB boot, dis_ucode_ldr) but let me comment on what hopefully is at least a step enabling further looking into: What I did so far based on your reply: - Removed quiet from GRUB_CMDLINE_LINUX_DEFAULT (which I could not find earlier) - In sysconfig yast, set the SECURE_BOOT to "no" - ran /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg (which I was already doing after changes before) - Rebooted - The system still hangs most of the time so it seems the SECURE_BOOT does not affect this - The "quiet remove" has an effect - I can now see messages when the system hangs. How can I deliver the messages to this report? I took a camera picture of the messages, but the result is too big for an attachment. Can I email it to your email (if so, what is it)? Or is there a log this stuff goes to? I see no clear errors there, except (this may be a message) "radeon: ... registered panic notifier". The last message is ""systemd-journald ... Received request to flush runtime journal from PID 1" Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c14 --- Comment #14 from Takashi Iwai --- (In reply to milan zimmermann from comment #12)

How can I deliver the messages to this report? I took a camera picture of the messages, but the result is too big for an attachment. Can I email it to your email (if so, what is it)? Or is there a log this stuff goes to?

It's not problem to attach a picture on Bugzilla. Use attachment. The size doesn't matter unless it's over 100MB or so :)

I see no clear errors there, except (this may be a message) "radeon: ... registered panic notifier". The last message is ""systemd-journald ... Received request to flush runtime journal from PID 1"

Maybe your first test with nomodeset wasn't performed properly? Could you retry with nomodeset boot option? This will result in the lower (or no) graphics with VESA fb, but it should be working at least. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c16 --- Comment #16 from milan zimmermann --- Created attachment 646863 --> http://bugzilla.opensuse.org/attachment.cgi?id=646863&action=edit Error message when boot hangs -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c18 --- Comment #18 from milan zimmermann --- Created attachment 646902 --> http://bugzilla.opensuse.org/attachment.cgi?id=646902&action=edit dmesg_after_first_successful_boot_with_nomodeset -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c20 --- Comment #20 from Takashi Iwai --- (In reply to milan zimmermann from comment #17)

I tested with nomodeset making sure to mkconfig and that it appears in /boot/grub2/grub.cfg. It did not help, failed to boot 5 times in 6 tries. In the succesful boot, the screen resolution was lower, the display narrower and stretched, all indications that the nomodeset did kick in.

But interestingly, in dmesg with the nomodeset, there is an error:

[ 3.167331] pata_jmicron 0000:05:00.1: enabling device (0000 -> 0001) [ 3.167740] [drm] VGACON disable radeon kernel modesetting. [ 3.167758] [drm:radeon_init [radeon]] *ERROR* No UMS support in radeon module!

This is OK, the expected result.

I am attaching the full dmesg.

Overall, from some 1000 boots or so I did during the last week, the only reliably working setting was when I boot using USB and in F5 (Kernel), set "No ACPI" . But setting acpi=off in the bootloader does not have the same effect as I noted.

ACPI=off supposedly disables some devices indirectly, so this might help avoiding the bad point.

Not sure where to take it next, but thanks very much for your help so far.

BTW, Would you have some idea how to find what actual setting is set when I select "No ACPI" in the USB boot in F5 (Kernel)?

You can take a look at /proc/cmdline. Judging from the boot screen you attached in comment 16, this doesn't seem like a crash of any driver. Now I read through the kernel log, the possible hit after the last dying message is acpi-cpufreq. Could you try to blacklist it, e.g. adding the following line to /etc/modprobe.d/99-local.conf? blacklist acpi-cpufreq Then reboot and retest. Check dmesg output to verify whether acpi-cpufreq If a message with "acpi-cpufreq" appears, the blacklist didn't work -- as a temporary test, just remove the module from /lib/modules/$VERSION/kernel/drivers/cpufreq directory. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c22 --- Comment #22 from milan zimmermann --- so after the last set of experiments, I realized I did not look at dmesg for a while. There is a ACPI warning pasted below. Interestingly, it is just below to where the system would normally hang! Googling for the ACPI warning, there are some recent reports around it, at least the first two reporting hang or lockup of X: https://bugs.freedesktop.org/attachment.cgi?id=117111&action=edit https://lists.debian.org/debian-x/2015/06/msg00095.html https://forums.gentoo.org/viewtopic-p-7781756.html?sid=0c2eb14b626e7550d3943... -------- [ 3.782610] hid-generic 0003:046D:C01B.0002: input,hidraw1: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:13.0-1/input0 [ 3.798793] input: HOLTEK USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb4/4-5/4-5:1.1/0003:04D9:A085.0003/i nput/input3 [ 3.849456] hid-generic 0003:04D9:A085.0003: input,hidraw2: USB HID v1.10 Mouse [HOLTEK USB Keyboard] on usb-0000 :00:12.0-5/input1 [ 4.002437] Switched to clocksource tsc [ 4.376601] [drm] ib test on ring 5 succeeded [ 4.377028] [drm] Radeon Display Connectors [ 4.377034] [drm] Connector 0: [ 4.377039] [drm] VGA-1 [ 4.377044] [drm] DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c [ 4.377051] [drm] Encoders: [ 4.377055] [drm] CRT1: INTERNAL_KLDSCP_DAC1 [ 4.377059] [drm] Connector 1: [ 4.377063] [drm] DVI-D-1 [ 4.377067] [drm] HPD1 [ 4.377072] [drm] DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c [ 4.377079] [drm] Encoders: [ 4.377083] [drm] DFP1: INTERNAL_KLDSCP_LVTMA [ 4.441898] [drm] fb mappable at 0xD0359000 [ 4.441906] [drm] vram apper at 0xD0000000 [ 4.441911] [drm] size 8294400 [ 4.441915] [drm] fb depth is 24 [ 4.441919] [drm] pitch is 7680 [ 4.442036] fbcon: radeondrmfb (fb0) is primary device [ 4.447652] Console: switching to colour frame buffer device 240x67 [ 4.454777] radeon 0000:01:05.0: fb0: radeondrmfb frame buffer device [ 4.454813] radeon 0000:01:05.0: registered panic notifier [ 4.457650] [drm] Initialized radeon 2.42.0 20080528 for 0000:01:05.0 on minor 0 [ 4.476789] PM: Starting manual resume from disk [ 4.476827] PM: Hibernation image partition 8:2 present [ 4.476828] PM: Looking for hibernation image. [ 4.477083] PM: Image not found (code -22) [ 4.477087] PM: Hibernation image not present or could not be loaded. [ 4.493388] BTRFS info (device sda1): use ssd allocation scheme [ 4.493433] BTRFS info (device sda1): disk space caching is enabled [ 4.692211] scsi 10:0:0:0: Direct-Access Lexar USB Flash Drive 1100 PQ: 0 ANSI: 6 [ 4.692509] sd 10:0:0:0: Attached scsi generic sg2 type 0 [ 4.694186] sd 10:0:0:0: [sdb] 15671296 512-byte logical blocks: (8.02 GB/7.47 GiB) [ 4.694949] sd 10:0:0:0: [sdb] Write Protect is off [ 4.694987] sd 10:0:0:0: [sdb] Mode Sense: 22 00 00 00 [ 4.695717] sd 10:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 4.700850] sdb: sdb1 sdb2 [ 4.705350] sd 10:0:0:0: [sdb] Attached SCSI removable disk [ 4.891513] systemd-journald[205]: Received SIGTERM from PID 1 (systemd). [ 5.058420] BTRFS info (device sda1): disk space caching is enabled [ 5.079292] random: nonblocking pool is initialized [ 5.085206] systemd-journald[448]: Received request to flush runtime journal from PID 1 ^^^^^^ INTERESTINGLY, THIS IS ROUGHLY WHERE THE PROCESS TYPICALLY HANGS ^^^^^ [ 5.159629] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.162267] wmi: Mapper loaded [ 5.165034] ACPI Warning: SystemIO range 0x0000000000000B00-0x0000000000000B07 conflicts with OpRegion 0x0000000000000B00-0x0000000000000B0F (\SOR1) (20150410/utaddress-254) [ 5.165047] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 5.165473] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05 [ 5.165540] sp5100_tco: PCI Revision ID: 0x42 [ 5.165575] sp5100_tco: Using 0xfed80b00 for watchdog MMIO address [ 5.165589] sp5100_tco: Last reboot was not triggered by watchdog. [ 5.166422] sp5100_tco: initialized (0xffffc900018c2b00). heartbeat=60 sec (nowayout=0) [ 5.174157] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input4 [ 5.174426] ACPI: Power Button [PWRB] [ 5.174534] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input5 [ 5.174582] ACPI: Power Button [PWRF] [ 5.198804] xhci_hcd 0000:02:00.0: xHCI Host Controller li -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c24 --- Comment #24 from Takashi Iwai --- Sorry I've been on vacation. And, yes, now the best option is to try beta1 installation. If this still shows the same issue, please try the kernel in OBS Kernel:openSUSE-42.1 repo, which is based on 4.1.8 now. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c26 --- Comment #26 from Takashi Iwai --- (In reply to milan zimmermann from comment #25)

Thanks for following up.

The issue is the same after update to 42.1 Beta.

After many long searches I came to a hard to prove and (of course quite possibly incorrect) conclusion this has something to do with the Audio support in the kernel; there are dmesg like

snd_hda_codec_hdmi hdaudioC1D0: HDMI ATI/AMD: no speaker allocation for ELD

just at the time the system normally hangs.

This is an utterly harmless message, found normally when plugging with a monitor without a speaker, and it must be just coincidence that this is seen at last. I can say it because I am the upstream maintainer of sound subsystem :) What we really need is to figure out whether this is really a kernel hang. If yes, what kind of hang. Since you don't get any kernel Oops or panic message, it doesn't look like a normal kernel hang due to a kernel bug, but either a hardware hang (hardware defect or hang by a driver bug) or some bad task blocking the whole system. As acpi=off seems curing, the odd is more to the former. If so, it's tough to figure out. You need to start from a minimal system that works reliably by disabling the hardware components as much as possible, then enable piece by piece until it hits the issue again. Or you can try older kernels. For example, 3.11.x kernel in openSUSE-13.1, 3.12.x for SLE12, 3.16.x for openSUSE-13.2. If any older kernel works, we may try bisection to spot out the regression. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c28 --- Comment #28 from milan zimmermann --- So I installed the kernel from this repo: http://download.opensuse.org/repositories/Kernel:/openSUSE-13.2/standard the version is 3.16.7-100. Out of 10 boots (5 power off/on, 5 resets) it booted 10 times - 100%. (Going back to 42.1 kernel failed 3 out of 4). So I am now reasonably convinced it is a Kernel regression somewhere along the way. Attaching the dmesg from the 3.16.7-100 boot. Can I provide more info? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c30 --- Comment #30 from Takashi Iwai --- There are a few other kernels in OBS home:tiwai:kernel:3.17, home:tiwai:kernel:3.18, ... up to :4.0. You can try it until you hit the same problem. 3.17.x might be already problematic, from my very wild guess. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c32 --- Comment #32 from milan zimmermann --- (In reply to milan zimmermann from comment #31)

Just added http://download.opensuse.org/repositories/home:/tiwai:/kernel:/3.17/standard... as a repo, will switch the kernel and test it.

This kernel (3.17.6-1.g12b7bf1-desktop) booted 5 times out of five. That is probably enough text for this one, but will try a few more tomorrow. Dmesg attached. Will test 3.18 and 3.19 next - tomorrow after I get some work done, it is way after midnight here, will report here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 milan zimmermann changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #649549|dmesg from an older kernel. |dmesg from 3.16.7. So far, description|So far, the system boots |the system boots |successfully 100% of time |successfully 100% of time |with this kernel. |with this kernel. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c35 --- Comment #35 from milan zimmermann --- Created attachment 649706 --> http://bugzilla.opensuse.org/attachment.cgi?id=649706&action=edit dmesg from 3.18, the boot problems start here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c37 --- Comment #37 from milan zimmermann --- As a note, I deducted in the past, if the boot hangs, it corresponds to the time between 4 seconds and 6 seconds into dmesg. Interestingly, the 3.18 boot has this at that time: [ 4.106244] radeon 0000:01:05.0: registered panic notifier [ 4.112156] [drm] Initialized radeon 2.40.0 20080528 for 0000:01:05.0 on minor 0 [ 6.116215] floppy0: no floppy controllers found [ 6.139987] PM: Starting manual resume from disk Not sure that means anything though. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c39 --- Comment #39 from milan zimmermann --- I should summarize as I added some noise: - From all I can tell, the bug of unsuccessful boots has been introduced between kernels 3.17.6-1.g12b7bf1-desktop (works) and 3.18.6-1.1 (often hangs on boot). Dmesg attached. Thanks -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c41 --- Comment #41 from Takashi Iwai --- Sorry for the long latency of my response, as I've been traveling often for now. And thanks for narrowing down the range. Up from this point, we need more detailed tests from you: namely, compiling the kernel by yourself and examining the regression point via git bisect. As a start, install gcc, git-core and make packages if not yet present. Then clone the linux kernel git repo from upstream % mkdir /somewhere/kernel % cd /somewhere/kernel % git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git where /somewhere/kernel is any directory you prefer to put the repo and build. Note that this will fetch many data from the net. Then, boot with 3.17 kernel. Then, checkout v3.17, and do localconfig. % cd /somewhere/kernel/linux % git checkout v3.17 % make localconfig This might ask you something, and usually just press RETURN for each to choose the default. In the end, you'll have ".config" file matching with your running system. Edit this file via a text editor and change CONFIG_LOCALVERSION to a different value, e.g. "-bisect". After the configuration, do make. Better to with -j4 or such to build parallel. % make -j4 Once when everything is done, install it. Be root, then run make install. % su % make install Then reboot to this kernel, and confirm that it still works. The next is to check 3.18 that should be broken. % cd /somewhere/kernel/linux % git checkout v3.18 At this point, better to backup your old config % cp .config .config.3.17 and run make oldconfig again, adjust to 3.18. % make oldconfig Rebuild, reinstall, and retest. Confirm that 3.18 is really broken. Now we start git bisection. Boot any good working kernel. Maybe good to backup 3.18 config again before starting git bisection. % cd /somewhere/kernel % cp .config .config.3.18 OK, here we go. % git bisect start % git bisect good v3.17 % git bisect bad v3.18 This will lead you some commit between 3.17 and 3.18. Do make localconfig, rebuild, reinstall, and retest. If this test kernel works, give "git bisect good". Otherwise reboot with a working kernel, and give "git bisect bad". Then git will lead you to yet another point, and you rebuild, reinstall and retest. Repeat this until you hit the regression point. You can see the current bisection process at any time via "git bisect log". Note that git bisect sometimes jumps to an old kernel version suddenly. This is normal, as the commit is actually based on that version. So, at installing and testing the kernel, be careful which kernel you're testing. While testing, you can remove the old installed kernels. I don't know a good default way for that, but you can just remove /lib/modules/$VERSION directory as well as /boot/vmlinuz-$VERSION, /boot/initrd-$VERSION and /boot/System.map-$VERSION. After cleaning up, run below as root /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg to refresh the GRUB menu. Oh, and after the bisection is done, run "git bisect reset" to reset the bisection state. Don't forget to save your bisection procedure from "git bisect log" beforehand. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=944659 http://bugzilla.opensuse.org/show_bug.cgi?id=944659#c43 Stephan Kulow changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |coolo@suse.com Flags| |SHIP_STOPPER- --- Comment #43 from Stephan Kulow --- this problem has been around for a while and only seems to manifest on milan's machine (as far as we know). -- You are receiving this mail because: You are on the CC list for the bug.