http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c2 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |toomas.aas@raad.tartu.ee Flags| |needinfo?(toomas.aas@raad.t | |artu.ee) --- Comment #2 from Oliver Neukum <oneukum@suse.com> --- (In reply to Takashi Iwai from comment #1)

(In reply to Toomas Aas from comment #0)

...

Interesting thing is that this only happens if the system has 10 GB RAM. My system has 2x1GB plus 2x4GB DIMMs, and when I remove one of the 4 GB DIMMs (thus reducing the amount of memory to 6 GB) the problem does not appear. To rule out the possibility of bad DIMM I have run memtest86 for several hours without finding any problems, also the system is rock solid with 10 GB RAM when I remove the USB wireless adapter and use wired Ethernet.

So this implies that the host side influences on the problem. Oliver, any clue? Some DMA issue?

It does look like DMA. We need to determine which HC is used. Please provide the output of "lsusb -t" -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c8 --- Comment #8 from Toomas Aas <toomas.aas@eesti.ee> --- I wasn't using this wifi adapter in the meantime, but took it out again few days ago. Now that the kernel is updated to: 4.1.31-30-default #1 SMP PREEMPT there is this additional line in dmesg output when the problem happens: ieee80211 phy0: rt2x00usb_watchdog_tx_dma: Warning - TX queue 0 DMA timed out, invoke forced forced reset -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c11 Joerg Roedel <jroedel@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(oneukum@suse.com) --- Comment #11 from Joerg Roedel <jroedel@suse.com> --- (In reply to Toomas Aas from comment #10)

Booting with kernel parameter 'iommu=soft' did not alleviate the problem.

Thanks for verifying this, Toomas. So the problem also shows up when no IOMMU is in use at all. Oliver, any other ideas? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c13 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(toomas.aas@eesti. | |ee) --- Comment #13 from Oliver Neukum <oneukum@suse.com> --- Created attachment 693344 --> http://bugzilla.opensuse.org/attachment.cgi?id=693344&action=edit limit DMA to 4GB Please apply this and provide dmesg -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c15 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED Flags| |needinfo?(toomas.aas@eesti. | |ee) --- Comment #15 from Oliver Neukum <oneukum@suse.com> --- Does the bug still happen? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c17 Toomas Aas <toomas.aas@eesti.ee> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(toomas.aas@eesti. | |ee) | --- Comment #17 from Toomas Aas <toomas.aas@eesti.ee> --- Unfortunately I have to report that the problem happened again today. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c24 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(toomas.aas@eesti. | |ee) --- Comment #24 from Oliver Neukum <oneukum@suse.com> --- (In reply to Toomas Aas from comment #23)

Created attachment 698959 [details] dmesg output with debugging enabled

Here is the dmesg output that I captured today. Unfortunately, by the time I noticed that the problem had appeared, very first error messages were no longer present in dmesg buffer and I couldn't retrieve them. Hopefully this is still helpful.

According to the logs scanning fails. Please try the kernel to be found at http://kernel.suse.com/packages/vanilla -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c32 Toomas Aas <toomas.aas@eesti.ee> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|Leap 42.1 |Leap 42.3 Flags|needinfo?(toomas.aas@eesti. | |ee) | --- Comment #32 from Toomas Aas <toomas.aas@eesti.ee> --- Bisecting took quite a long time, because I have no way to cause the problem to happen. With every new kernel I had to just wait (which could take up to 2 weeks). However, I have finally found the commit which resolves the problem. I have used the kernel with this commit successfully for more than a month. The commit that resolves the problem is this: commit e6cbd7f2efb433d717af72aa8510a9db6f7a7e05 Author: Mel Gorman <mgorman@techsingularity.net> Date: Thu Jul 28 15:46:50 2016 -0700 mm, page_alloc: remove fair zone allocation policy While the original bug report was submitted for Leap 42.1, the problem was still present in 42.2 and is also present in 42.3 with this kernel from the 'kernel-default' package: Linux susa 4.4.114-42-default #1 SMP Tue Feb 6 10:58:10 UTC 2018 (b6ee9ae) x86_64 x86_64 x86_64 GNU/Linux -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c33 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mgorman@suse.com Flags| |needinfo?(mgorman@suse.com) --- Comment #33 from Oliver Neukum <oneukum@suse.com> --- Mel, it looks like we are seeing an issue with MM acting up on machines with a lot of RAM in 42.3. I don't suppose we can take a patch that alters MM so fundamentally. Any possible work arounds? Creative gfp flags? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c35 --- Comment #35 from Mel Gorman <mgorman@suse.com> --- (In reply to Oliver Neukum from comment #33)

Mel, it looks like we are seeing an issue with MM acting up on machines with a lot of RAM in 42.3. I don't suppose we can take a patch that alters MM so fundamentally. Any possible work arounds? Creative gfp flags?

Note that I'm not very active in MM at the moment due to other responsibilities. However, I think the fact that it bisects to this particular commit may be a partial co-incidence. The nature of the error messages appear to be due to a failure when calling usb_control_msg has an error when called from the driver. usb_control_msg calls kmalloc(sizeof(struct usb_ctrlrequest), GFP_NOIO) so what is likely to be happening is that the kmalloc fails but it's not able to reclaim many pages due to the GFP context. The commit in question has a side-effect of causing kswapd to wake to reclaim from zones earlier than it does without the patch. Specifically, without the patch, zones are evenly used until kswapd is required. With the patch, the Normal zone in this case fills first, wakes kswapd, uses lower zones etc. As a side-effect, the early reclaim means memory is freed earlier and a GFP_NOIO call is more likely to successfully complete on an x86 system. Altering the amount of memory so that there is a normal zone could co-incidentally alter the timing of when kswapd wakes which may be why it's visible. Applying the patch is not without consequences. The patch does not work in isolation, it only works properly if all the dependent patches are included that move all the LRU lists to the node and that is a non-trivial backport that would cause KABI issues if it was included which forces it to be 42.3-specific. If the patch is included on its own, it'll introduce page age inversion issues and so introduce regressions that are easy to detect but very difficult to isolate as the root cause. Any tuning option from the MM side requires other patches. However, increasing min_free_kbytes *may* mitigate the problem so it is worth trying but it may also just delay the problem for longer. The other option is a 42.3-specific hack that should not be forward-ported that specifies __GFP_HIGH in usb_control_msg as a failure to allocate there can result in complete failure of a driver. That would allow the drivers to dip into the page allocation reserves similar to what can happen in IRQ context (which is not the context here) and hope kswapd catches up before the reserve is depleted. It might be lower risk overall. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c37 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CONFIRMED |IN_PROGRESS Flags| |needinfo?(toomas.aas@eesti. | |ee) --- Comment #37 from Oliver Neukum <oneukum@suse.com> --- A test kernel is abuilding at https://build.opensuse.org/project/monitor?project=home%3Aoneukum%3Absc95923... Please test -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c39 --- Comment #39 from Takashi Iwai <tiwai@suse.com> --- (In reply to Toomas Aas from comment #38)

Created attachment 762405 [details] Logs from booting with kernel-default-4.4.117-1.1.gd04fda6

Hi Oliver!

I installed the kernel from comment #37, but this causes a side effect. The graphical login screen is frozen. The mouse cursor moves, but clicking mouse buttons or typing on keyboard has no effect. However, I can Ctrl+Alt+F2 to text console and keyboard works there. But this probem means that I cannot do any longer term testing with this kernel.

Attached are logs from booting the new kernel and previous 4.4.114 kernel.

Hrm, there is anything wrong in the kernel logs, and it's radeon driver that was overridden by drm-kmp, so it should be same in both cases. In anyway, try to test with nomodeset boot option temporarily just for checking the WiFi behavior. Also, later on, you can try to uninstall drm-kmp-default and see whether the graphics issue persists. If not, you can keep it uninstalled. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c41 --- Comment #41 from Takashi Iwai <tiwai@suse.com> --- Also Oliver, could you rebase the test kernel with the latest Leap 42.3 KOTD and rebuild? The affecting patches that broke drm-kmp were reverted, so now it should work better. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c43 Toomas Aas <toomas.aas@eesti.ee> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(toomas.aas@eesti. | |ee) | --- Comment #43 from Toomas Aas <toomas.aas@eesti.ee> --- I apologize for lack of feedback. During past weeks I've been travelling a lot and haven't been able to do enough testing. I did get the system working with kernel from comment #37 after uninstalling drm-kmp-default. But after only two days I left for vacation, so there wasn't enough time to determine whether the main problem of wifi connection dropping has been resolved. Today I installed the kernel from comment #42 and also reinstalled drm-kmp-default. Initial result was the same as with kernel from comment #37 - login screen accepts no input. After uninstalling drm-kmp-default, the problem with login screen was again resolved. However, after just few minutes wifi connection dropped with the message: ieee80211 phy0: rt2x00usb_vendor_request: Error - Vendor Request 0x07 failed for offset 0x1700 with error -110 I am now running with kernel from comment #37 again, but I think from the point of view of resolving the wifi problem it is the same as kernel from comment #42, so it should be only a matter of time until the problem happens again? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c45 Oliver Neukum <oneukum@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(toomas.aas@eesti. | |ee) --- Comment #45 from Oliver Neukum <oneukum@suse.com> --- Well, the simple fix does not work, we cannot take the upstream fix, so it is time to get creative. Please test the kernel to be found at: https://build.opensuse.org/project/monitor?project=home%3Aoneukum%3Abnc95923... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 http://bugzilla.opensuse.org/show_bug.cgi?id=959230#c48 Toomas Aas <toomas.aas@eesti.ee> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(toomas.aas@eesti. | |ee) | --- Comment #48 from Toomas Aas <toomas.aas@eesti.ee> --- The problem is still present with kernel from comment #47. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=959230 Toomas Aas <toomas.aas@eesti.ee> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(toomas.aas@eesti. | |ee) | -- You are receiving this mail because: You are on the CC list for the bug.