http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c1 Frank Kruger changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fkrueger@mailbox.org --- Comment #1 from Frank Kruger --- I can confirm your observation. In fact, as of Build 603.1 with kernel-default-5.3.18-lp152.5.3.x86_64 I am experiencing a kernel panic, the former works fine. Current kernel from https://download.opensuse.org/repositories/Kernel:/openSUSE-15.2/standard/ does not work either (actually it only worked until kernel-default-5.3.18-lp152.20.* from this repo). In addition, kernel-default from Tumbleweed and the stable repo works fine on this machine. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c3 --- Comment #3 from Frank Kruger --- Created attachment 832797 --> http://bugzilla.opensuse.org/attachment.cgi?id=832797&action=edit hwinfo output -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c5 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nwr10cst-oslnx@yahoo.com, | |tiwai@suse.com Flags| |needinfo?(nwr10cst-oslnx@ya | |hoo.com) --- Comment #5 from Takashi Iwai --- (In reply to Neil Rickert from comment #0)

Booting with the previous kernel (5.3.18-lp152.4-default) is fine. After booting to the good kernel, I tried running "mkinitrd". But that did not fix the problem.

Could you boot with the previous running kernel, and get hwinfo, attach the output? Also give the dmesg output from the working kernel, too. Also, at best, if you get a screenshot of the kernel panic, take it, too. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c7 Neil Rickert changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nwr10cst-oslnx@ya | |hoo.com) | --- Comment #7 from Neil Rickert --- Created attachment 832801 --> http://bugzilla.opensuse.org/attachment.cgi?id=832801&action=edit dmesg output using previous kernel I have attached "hwinfo" and "dmesg" output, taken while using the previous kernel. I have since installed kernel "5.5.9-3.ge9e359d-default" from the kernels repo. And it boots fine with that kernel. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c9 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |giovanni.gherdovich@suse.co | |m --- Comment #9 from Takashi Iwai --- Adding Giovanni to Cc, as he has a better clue about the code path found in Oops in comment 2. FWIW, the recent patches (frequency invariance on ATOM, etc) have been added after the previous working kernel (5.3.18-lp152.4-default) for the original report. Though, Neil couldn't get the kernel panic message, so it's not clear whether it's the identical cause of panic in comment 2. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c12 Neil Rickert changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nwr10cst-oslnx@ya | |hoo.com) | --- Comment #12 from Neil Rickert ---

Could you test openSUSE-15.2 KOTD kernel?

I installed "vmlinuz-5.3.18-lp152.43.gb955dad-default" from that repo. Unfortunately, it does not boot -- exactly the same problem as reported in this bug. Likewise "vmlinuz-5.3.18-lp152.7-default" from today's update does not boot. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 Frank Kruger changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fkrueger@mailbox. | |org) | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c21 --- Comment #21 from Neil Rickert --- I have already added "oldest" to the list of kernels to be kept. That's to keep the kernel that works. At present the purge-kernels service isn't actually running, so I'm managing these manually anyway. I have added those two test repos. They don't yet seem to be ready. I'll check again later. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c23 --- Comment #23 from Neil Rickert --- My results are similar to those of Frank. Kernel "vmlinuz-5.3.18-lp152.1.g90f4f41-default", which is from home:tiwai:bsc1166664-p1 does not boot at all (same behavior as before). This is as you expected. Kernel "vmlinuz-5.3.18-lp152.1.gb5606c0-default", from home:tiwai:bsc1166664-p2 boots just fine, and is up and running now (not the computer I am using to report this). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c25 --- Comment #25 from Frank Kruger --- (In reply to Takashi Iwai from comment #24)

Thanks for quick testing! This is a good step forward. It points that the regression is likely caused by the cpufreq patchset.

I'm building another test kernel in OBS home:tiwai:bsc1166664-p3 repo. This is based on the very latest openSUSE-15.2 git branch with the revert of suspicious patches. Please check whether this works later.

Your most recent kernel-default-5.3.18-lp152.1.1.g477312f.x86_64 works fine and solves the issue for me. Thx. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c27 Neil Rickert changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #27 from Neil Rickert --- Yes, 5.3.18-lp152.1.g477312f-default also works just fine here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bpetkov@suse.com, | |mgorman@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c31 Giovanni Gherdovich changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(nwr10cst-oslnx@ya | |hoo.com) --- Comment #31 from Giovanni Gherdovich --- The code crashes likely because it fails to correctly parse the value indicating the turbo frequecy of the processor; it incorrectly reads it as zero and somewhere down the line a division by zero is made. I am checking the documentation for the relevant model-specific registers of these two processors, but it would be of great help if I can get more data out of these two machines: Frank: according to the hwinfo from comment 3 you have an "Acer Aspire 3820T" laptop with a "Core i5-430M" cpu from 2010 (micro-arch codename: Westmere); can you share the output of this command: turbostat --interval 1 sleep 0 this should output the content of a number of power management-related registers and capabilities, that would help pinpoint what's the correct way to parse the values. Neil: according to the hwinfo from comment 6 you have a "Dell Inspiron N5010" laptop with a "Core i3-370M" cpu from 2010 (micro-arch codename: Westmere as well). According to the Intel product list this cpu doesn't have any Turbo Boost capability, and the crashing code fails to understand that. I'd also like to see the output you get from turbostat --interval 1 sleep 0 Thanks again to you both for testing early and reporting this problem. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c33 Neil Rickert changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nwr10cst-oslnx@ya | |hoo.com) | --- Comment #33 from Neil Rickert --- Created attachment 834204 --> http://bugzilla.opensuse.org/attachment.cgi?id=834204&action=edit Requested output from: turbostat --interval 1 sleep 0 Attaching a transcript ("typescript" file from "script" command) showing the output requested. And yes, this is a 2010 Dell laptop N5010. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c35 --- Comment #35 from Giovanni Gherdovich --- Thanks Neil and Frank! The mistake made in the code was to assume that all machines have at least 4 physical cores. Testing was done on server-class machines but also on a mobile-type machine called "Dell Wyse 5010" from 2016, which happens to have 4 cores so this problem wasn't noticed. The machines mentioned in this bug both have 2 physical cores, and the problem manifests; I'm happy this got caught before release. Some more details: This particular piece of code needs to know the "average maximum" speed a CPU can go, in order to normalize the frequency value at any point in time and understand if the processor is running at capacity or not. "Average maximum" is a bit of a guessing: true maximum clock speed would be the turbo value with only 1 core active, but that might not be available (what if more cores are active at the same time?). Testing shows that the turbo frequency sustainable by 4 cores simultaneously (4 cores turbo) is a good compromise in this formula, in term of application performances, between base frequency (no turbo) and 1 core turbo. So what happens when the OS asks the hardware "how fast can you push 4 cores at the same time?" but the CPU has only 2 cores? The answer is zero megahertz, as it can be seen the MSR_TURBO_RATIO_LIMIT register eg. in Frank's Aspire machine: MSR_TURBO_RATIO_LIMIT = 0x00001313 4 cores turbo = (0x00001313 >> 24) & 0xFF = 0 Later on, the first time a normalization current_freq / 4cores_turbo is made, that's a division by zero and we get the panic. The fix in this case is to take the 1 core turbo as normalizing factor. The other choice would be to use the base frequency (no turbo at all), but using base freq for normalization means that the ratio becomes current_freq / base_freq which is an over-estimation, which leads the frequency governor to think the CPU is more loaded than it really is and raise the frequency a bit too aggressively. This is tolerable in performance-oriented servers but inappropriate on small machines with 2 cores. To be noted that this choice (normalizing factor = 1 core turbo freq) works also on Neil's Inspiron machine, which doesn't support turbo at all: on that platform the only choice is to use base_freq, and indeed querying the hardware for 1 core turbo returns the base frequency, as is shown in the turbostat output: cpu2: MSR_PLATFORM_INFO: 0x70020011200 7 * 133.3 = 933.3 MHz max efficiency frequency 18 * 133.3 = 2399.9 MHz base frequency <---- base frequency cpu2: MSR_TURBO_RATIO_LIMIT: 0x00001212 18 * 133.3 = 2399.9 MHz max turbo 2 active cores 18 * 133.3 = 2399.9 MHz max turbo 1 active cores <---- base frequency again I'll let when the fix is ready. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c37 --- Comment #37 from Giovanni Gherdovich --- (In reply to Stakanov Schufter from comment #36)

Created attachment 834900 [details] Turbostat output of a Thinkpad X201 Lenovo

As I was hit by this too, I join the output of turbostat --interval 1 sleep 0

for a Lenovo Thinkpad X201

Thank you very much Stakanov for adding info to this bug. Your report confirms that the root cause is correctly identified, i.e. this kernel is broken on machines with less than 4 physical cores. From your turbostat output I learn that your machine has two physical cores: cpu0: MSR_TURBO_RATIO_LIMIT: 0x00001416 20 * 133.3 = 2666.6 MHz max turbo 2 active cores 22 * 133.3 = 2933.3 MHz max turbo 1 active cores (that makes four threads with hyper-threading). Let me reiterate that I'm writing a fix for this and a test kernel will soon be available for you to verify that the problem is resolved. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c39 Giovanni Gherdovich changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(nwr10cst-oslnx@ya | |hoo.com) --- Comment #39 from Giovanni Gherdovich --- Hello Neil and Frank, the fix is ready and committed in the openSUSE 15.2 kernel tree. I'll let you know when a test build is available. Question for you two: since I'll have to send this patch to the upstream Linux kernel as well, are you OK with being credited as "Reportd-by: name <email address>" in the git commit message? I'm asking because sometimes people prefer not expose their email address for privacy concerns. Let me know, I just thought it was fair to credit you two as you found the bug but I'm happy to comply if you prefer otherwise. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c41 --- Comment #41 from Frank Kruger --- (In reply to Giovanni Gherdovich from comment #40)

see comment 39, need-info from Frank as well.

I prefer not to be mentioned. By the way, why is this bug restricted to 5.3.18 and does not show up in the current TW/Kernel:stable version? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c43 Giovanni Gherdovich changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fkrueger@mailbox. | |org) | --- Comment #43 from Giovanni Gherdovich --- (In reply to Frank Kruger from comment #42)

(In reply to Giovanni Gherdovich from comment #39)

...

Hello Neil and Frank,

the fix is ready and committed in the openSUSE 15.2 kernel tree. I'll let you know when a test build is available.

Do you mean kernel-default-5.3.18-lp152.54.1.ga52aaa2.x86_64 from Kernel:openSUSE-15.2? This does not boot either. After "Loading intial ramdisk" it hangs with a black screen (no messages at all).

Yeah if I read that version tag correctly, it's from commit a52aaa2bcea0dca65a0459972c11ab86a800550b which should have the fix (the fix itself is 949f5c145e98b217570829dcff769d7dbecd9343 from https://github.com/SUSE/kernel-source/tree/openSUSE-15.2 ). I'm looking at it, thanks for testing. (In reply to Frank Kruger from comment #41)

(In reply to Giovanni Gherdovich from comment #40)

...

see comment 39, need-info from Frank as well.

I prefer not to be mentioned.

Noted. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c45 --- Comment #45 from Frank Kruger --- Just to be on the safe side, Giovanni, could you please provide the KOTD without your patches/fixes? Thx. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c47 --- Comment #47 from Neil Rickert --- I've just tested kernel 5.3.18-lp152.54.ga52aaa2-default This is booting fine for me. I have booted twice since installing that kernel. I'm not seeing any problems. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c49 --- Comment #49 from Giovanni Gherdovich --- (In reply to Frank Kruger from comment #48)

(In reply to Frank Kruger from comment #44)

...

Let me summarize the current situation (at least for me): Kernel from Kernel:openSUSE-15.1 und TW/Kernel:stable work fine, the KOTD from Kernel:openSUSE-15.2 including your fix does not. The kernel from comment 24 without your patches works fine.

The aforementioned results were obtained by using Graphic Mode=Discrete. Changing it to "switchable" the KOTD (kernel-default-5.3.18-lp152.54.1.ga52aaa2.x86_64) boots fine. Any idea?

Uhm no, it doesn't ring any bell. It could be a separate issue. I'm adding Patrik Jakobsson and Thomas Zimmermann for the graphics part. Patrick, Thomas: Frank from above comment can't boot our latest Leap 15.2 KOTD on his 2010 "Acer Aspire 3820T" which has an Intel "Core i5-430M" cpu (Westmere) if he selects "Graphic Mode=Discrete" from the BIOS config menu. See the full hwinfo for that machine in comment 3. According to hwinfo, the graphics card shows up as an "ATI Park [Mobility Radeon HD 5430/5450/5470]". Would you have any idea as of why? For context: this bug started from a kernel panic in the function arch_scale_freq_tick from the kernel source file arch/x86/kernel/smpboot.c , and this new issue came up when verifying whether the fix works. It may or may not be a different problem. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c50 --- Comment #50 from Frank Kruger --- It seems that I am also hit by bug 1169038. In fact, using the recent kernel from https://download.opensuse.org/repositories/home:/tiwai:/bsc1169038/standard/ (i.e. kernel-default-5.3.18-lp152.1.1.g5ac1e9a.x86_64), which also contains Giovanni's fix, boots fine (independent on the graphic mode). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c52 --- Comment #52 from Frank Kruger --- (In reply to Takashi Iwai from comment #51)

Yes, there is another known recent regression on SLE15-SP2 / Leap 15.2 kernel due to the bad padding in pci_fixup table. Some of you might have hit this. The bad commit was introduced recently, and the fix is on its way (but not merged / released yet).

So, please wait for a bit, hopefully it'll get merged after Easter holidays.

KOTD from https://download.opensuse.org/repositories/Kernel:/openSUSE-15.2/standard/ (kernel-default-5.3.18-lp152.55.1.gd43ae1f.x86_64) works for me. Thx. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c55 Andras FEHER changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- --- Comment #55 from Andras FEHER --- (In reply to Jiri Slaby from comment #54)

*** Bug 1169608 has been marked as a duplicate of this bug. ***

openSUSE-Leap-15.2-DVD-x86_64-Build632.2-Media.iso still Install Kernel Panic Created attachment 835885 [details] scr picture Asus X52F laptop Boot from USB -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1166664 http://bugzilla.opensuse.org/show_bug.cgi?id=1166664#c57 --- Comment #57 from Neil Rickert --- Build 638.2 is now out. It has an updated kernel (5.3.18-lp152.11-default). My laptop is booting fine with that kernel. To be clear -- I did not do a new install. Instead, I updated (with "zypper dup") and then tried booting to the new kernel. And that works. If you were having install problems, you should try with this updated build 638.2. I think your system will now boot the installer with this build. -- You are receiving this mail because: You are on the CC list for the bug.