[opensuse-kernel] Heavy performance loss since 4.16.0-rc6
Hello all, I was recommended to post my observation to this mailing list; hope that this is appropriate. Since mid-March, my Tumbleweed suffers a drastic performance loss. I observed this during my work on MAME (the emulator framework) which is an application with high CPU load. For benchmark purposes, MAME offers a command line option that suppresses screen output and turns off real-time synchronization so that the emulator core is running at the maximum speed. Typically you should expect a multiple of the real speed, expressed by a percentage of far above 100%. At some time in mid-March, the benchmark speed for a selected system dropped from 900% to 300%, for another emulated system from 450% to 150%, which is already too low to ensure a full emulation speed with screen output. To rule out any effects from the MAME application, I have been using a version that I built almost a year ago, leaving it unchanged. The described slowdown occurs on three separate desktop PCs, which means that a hardware issue is unlikely. All PCs have a Core-i7 architecture. Interestingly, on two laptops that I could use to test, the effect does not occur. Consequently, this means that now my old laptop from 2012 with a Core i5-3320M@2.6 GHz runs MAME faster than my Core i7-6700K@4.0 GHz. The slowdown also occurs in other applications. Notably, the Leawo Bluray Player that runs in WINE is not able to sustain the frame rate any longer, delivering chopped replay, which was perfect before. During the last days I tried to bisect the kernel source from Github in order to isolate the point where the effect started (remote origin: git@github.com:openSUSE/kernel.git). Interestingly, the effect did not show up until 4.16.0-rc6, while that same desktop PC already showed the effect during the distro upgrade around 4.15.13 (which was in March). The result is: bad: 646d6e114288 Update to 4.16-rc6. good: aec6bb8e2770 - Refresh patches.suse/kernel-add-release-status-to-kernel-build.patch. If desired, I can give you a small bundle of the MAME executable to run the benchmark by yourself. Here is the effect: michael@capella:~/susekernel> uname -a Linux capella 4.16.0-rc5-1-default+ #26 SMP PREEMPT Sun Apr 22 14:26:13 CEST 2018 x86_64 x86_64 x86_64 GNU/Linux michael@capella:~/susekernel> git status HEAD detached at aec6bb8e2770 nothing to commit, working tree clean michael@capella:~/susekernel> cd ~/mame michael@capella:~/mame> ./mame64_0186 ti99_4a -bench 20 Average speed: 883.31% (19 seconds) michael@capella:~/mame> michael@capella:~/susekernel> uname -a Linux capella 4.16.0-rc6-1-default+ #27 SMP PREEMPT Sun Apr 22 16:17:52 CEST 2018 x86_64 x86_64 x86_64 GNU/Linux michael@capella:~/susekernel> git status HEAD detached at 646d6e114288 nothing to commit, working tree clean michael@capella:~/susekernel> cd ~/mame michael@capella:~/mame> ./mame64_0186 ti99_4a -bench 20 Average speed: 305.27% (19 seconds) michael@capella:~/mame> Greetings, Michael -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Le 23/04/2018 à 13:01, Michael Zapf a écrit :
Hello all,
I was recommended to post my observation to this mailing list; hope that this is appropriate.
Since mid-March, my Tumbleweed suffers a drastic performance loss.
isn't that related to kernel exploit like Spectre? just a guess jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Am 23.04.2018 um 13:30 schrieb jdd@dodin.org:
Le 23/04/2018 à 13:01, Michael Zapf a écrit :
Hello all,
I was recommended to post my observation to this mailing list; hope that this is appropriate.
Since mid-March, my Tumbleweed suffers a drastic performance loss.
isn't that related to kernel exploit like Spectre?
I just tried to boot rc6 with "nopti nospec" as parameters, but had no effect. Michael -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Monday, 23 April 2018 13:01 Michael Zapf wrote:
During the last days I tried to bisect the kernel source from Github in order to isolate the point where the effect started (remote origin: git@github.com:openSUSE/kernel.git). Interestingly, the effect did not show up until 4.16.0-rc6, while that same desktop PC already showed the effect during the distro upgrade around 4.15.13 (which was in March).
The result is:
bad: 646d6e114288 Update to 4.16-rc6. good: aec6bb8e2770 - Refresh patches.suse/kernel-add-release-status-to-kernel-build.patch.
Looks like there is a good chance the performance drop comes from some of the mainline commits between v4.16-rc5 and v4.16-rc6 which has been backported into stable-4.15 (before we switched to 4.16). The only problem is there is 62 of them... (the list is below). But if you got so far, you could identify a particular patch: - checkout commit 950fc49446f4 from the kernel-source repository (https://github.com/openSUSE/kernel-source) - build kernel-vanilla to make sure the issue is not caused by one of the SUSE specific patches - bisect throught the list of 4.15.10 - 4.15.13 patches in series.conf by commenting out second half of them etc. You could also bisect only across the list below (and perhaps also exclude apparently irrelevant ones) but then you risk that the series won't apply if you disable some patches but leave some which are lower. Michal Kubecek List of 4.15 stable backports of commits in v4.16-rc5..v4.16-rc6: patches.kernel.org/4.15.10-143-NFS-Fix-an-incorrect-type-in-struct-nfs_direc.patch patches.kernel.org/4.15.10-144-pNFS-Prevent-the-layout-header-refcount-going.patch patches.kernel.org/4.15.10-145-NFS-Fix-unstable-write-completion.patch patches.kernel.org/4.15.11-002-ASoC-sun4i-i2s-Fix-RX-slot-number-of-SUN8I.patch patches.kernel.org/4.15.11-003-ASoC-sgtl5000-Fix-suspend-resume.patch patches.kernel.org/4.15.11-004-ASoC-wm_adsp-For-TLV-controls-only-register-T.patch patches.kernel.org/4.15.11-005-ASoC-rt5651-Fix-regcache-sync-errors-on-resum.patch patches.kernel.org/4.15.11-006-usb-host-xhci-rcar-add-support-for-r8a77965.patch patches.kernel.org/4.15.11-007-xhci-Fix-front-USB-ports-on-ASUS-PRIME-B350M-.patch patches.kernel.org/4.15.11-008-xhci-fix-endpoint-context-tracer-output.patch patches.kernel.org/4.15.11-009-serial-sh-sci-prevent-lockup-on-full-TTY-buff.patch patches.kernel.org/4.15.11-010-tty-serial-atmel-add-new-version-check-for-us.patch patches.kernel.org/4.15.11-011-uas-fix-comparison-for-error-code.patch patches.kernel.org/4.15.11-012-staging-comedi-fix-comedi_nsamples_left.patch patches.kernel.org/4.15.11-013-staging-android-ashmem-Fix-lockdep-issue-duri.patch patches.kernel.org/4.15.11-014-scsi-sd_zbc-Fix-potential-memory-leak.patch patches.kernel.org/4.15.11-015-USB-storage-Add-JMicron-bridge-152d-2567-to-u.patch patches.kernel.org/4.15.11-016-usbip-vudc-fix-null-pointer-dereference-on-ud.patch patches.kernel.org/4.15.11-017-usb-quirks-add-control-message-delay-for-1b1c.patch patches.kernel.org/4.15.11-018-usb-usbmon-Read-text-within-supplied-buffer-s.patch patches.kernel.org/4.15.11-019-usb-gadget-f_fs-Fix-use-after-free-in-ffs_fs_.patch patches.kernel.org/4.15.11-020-usb-dwc3-Fix-lock-up-on-ID-change-during-syst.patch patches.kernel.org/4.15.11-021-serial-8250_pci-Add-Brainboxes-UC-260-4-port-.patch patches.kernel.org/4.15.11-022-serial-core-mark-port-as-initialized-in-autoc.patch patches.kernel.org/4.15.11-023-earlycon-add-reg-offset-to-physical-address-b.patch patches.kernel.org/4.15.11-024-dm-mpath-fix-passing-integrity-data.patch patches.kernel.org/4.15.12-001-x86-cpufeatures-Add-Intel-Total-Memory-Encryp.patch patches.kernel.org/4.15.12-002-x86-cpufeatures-Add-Intel-PCONFIG-cpufeature.patch patches.kernel.org/4.15.12-003-selftests-x86-entry_from_vm86-Exit-with-1-if-.patch patches.kernel.org/4.15.12-004-selftests-x86-entry_from_vm86-Add-test-cases-.patch patches.kernel.org/4.15.12-005-x86-vm86-32-Fix-POPF-emulation.patch patches.kernel.org/4.15.12-006-x86-speculation-objtool-Annotate-indirect-cal.patch patches.kernel.org/4.15.12-007-x86-speculation-Remove-Skylake-C2-from-Specul.patch patches.kernel.org/4.15.12-008-KVM-x86-Fix-device-passthrough-when-SME-is-ac.patch patches.kernel.org/4.15.12-009-x86-mm-Fix-vmalloc_fault-to-use-pXd_large.patch patches.kernel.org/4.15.12-010-parisc-Handle-case-where-flush_cache_range-is.patch patches.kernel.org/4.15.12-011-ALSA-pcm-Fix-UAF-in-snd_pcm_oss_get_formats.patch patches.kernel.org/4.15.12-012-ALSA-hda-Revert-power_save-option-default-val.patch patches.kernel.org/4.15.12-013-ALSA-seq-Fix-possible-UAF-in-snd_seq_check_qu.patch patches.kernel.org/4.15.12-014-ALSA-seq-Clear-client-entry-before-deleting-e.patch patches.kernel.org/4.15.12-015-drm-nouveau-bl-Fix-oops-on-driver-unbind.patch patches.kernel.org/4.15.12-016-drm-nouveau-mmu-ALIGN_DOWN-correct-variable.patch patches.kernel.org/4.15.12-017-drm-amdgpu-fix-prime-teardown-order.patch patches.kernel.org/4.15.12-018-drm-radeon-fix-prime-teardown-order.patch patches.kernel.org/4.15.12-019-drm-amdgpu-dce-Don-t-turn-off-DP-sink-when-di.patch patches.kernel.org/4.15.12-020-fs-Teach-path_connected-to-handle-nfs-filesys.patch patches.kernel.org/4.15.12-021-KVM-arm-arm64-Reduce-verbosity-of-KVM-init-lo.patch patches.kernel.org/4.15.12-022-KVM-arm-arm64-Reset-mapped-IRQs-on-VM-reset.patch patches.kernel.org/4.15.12-023-kvm-arm-arm64-vgic-v3-Tighten-synchronization.patch patches.kernel.org/4.15.12-024-KVM-arm-arm64-vgic-Don-t-populate-multiple-LR.patch patches.kernel.org/4.15.12-025-lock_parent-needs-to-recheck-if-dentry-got-__.patch patches.kernel.org/4.15.12-026-fs-aio-Add-explicit-RCU-grace-period-when-fre.patch patches.kernel.org/4.15.12-027-fs-aio-Use-RCU-accessors-for-kioctx_table-tab.patch patches.kernel.org/4.15.12-028-RDMAVT-Fix-synchronization-around-percpu_ref.patch patches.kernel.org/4.15.12-029-irqchip-gic-v3-its-Ensure-nr_ites-nr_lpis.patch patches.kernel.org/4.15.12-032-btrfs-Fix-NULL-pointer-exception-in-find_bio_.patch patches.kernel.org/4.15.12-033-btrfs-add-missing-initialization-in-btrfs_che.patch patches.kernel.org/4.15.12-041-scsi-qla2xxx-Fix-crashes-in-qla2x00_probe_one.patch patches.kernel.org/4.15.12-042-usb-dwc2-fix-STM32F7-USB-OTG-HS-compatible.patch patches.kernel.org/4.15.12-043-dt-bindings-usb-fix-the-STM32F7-DWC2-OTG-HS-c.patch patches.kernel.org/4.15.13-002-staging-android-ashmem-Fix-possible-deadlock-.patch patches.kernel.org/4.15.13-079-serial-8250_pci-Don-t-fail-on-multiport-card-.patch -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Just some random guesses, but...since you narrowed it down to x86 and cpu...though I can't really tell what many of these are due to truncation... Michal Kubecek wrote:
On Monday, 23 April 2018 13:01 Michael Zapf wrote:
During the last days I tried to bisect the kernel source from Github
total memory encryption? sounds expensive...
patches.kernel.org/4.15.12-001-x86-cpufeatures-Add-Intel-Total-Memory-Encryp.patch patches.kernel.org/4.15.12-002-x86-cpufeatures-Add-Intel-PCONFIG-cpufeature.patch
Not sure speculation fixes shouldn't be causing *that* much slowdown... If you'd even said 30%, (which would be bad enough, IMO), I might have thought it possible, but what you are seeing seems outside what I thought were bounds...
patches.kernel.org/4.15.12-006-x86-speculation-objtool-Annotate-indirect-cal.patch patches.kernel.org/4.15.12-007-x86-speculation-Remove-Skylake-C2-from-Specul.patch
I can't see why encryption of all memory would be turned on by default, or even if that's really what that patch does, but if it *did*, ... I'd guess that it would slow things down quite a bit, but **too much** to ever be enabled by default unless there was a bug or misconfig...but still... -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hi all, a new observation, maybe crucial to that problem. I continued my tests on a PC in my second apartment. There I noticed that the bisect location does not apply anymore: This second PC was fast for the formerly bad commit as well. Still, the normal installation from Tumbleweed was still slow. I rolled back the state of the test installation to try again. There I found: Once I install ucode-intel 20180312-3.1, the system slows down from 900% to 300% in the benchmark. Rolling back before that ucode package was installed, it gets fast again. Could this explain why my laptops are not affected, while my three PCs are all hit by this slowdown? It is hard to believe that my 2012 laptop with a Core i5-3320M now visibly outperforms my 2017 PC with a i7-7700K. Let alone that the dually installed Windows 10 on the same PC does not suffer from any slowdown at all (using the Windows build of MAME). Greetings, Michael -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Wed, Apr 25, 2018 at 09:46:32PM +0200, Michael Zapf wrote:
Hi all,
a new observation, maybe crucial to that problem.
I continued my tests on a PC in my second apartment. There I noticed that the bisect location does not apply anymore: This second PC was fast for the formerly bad commit as well. Still, the normal installation from Tumbleweed was still slow.
I rolled back the state of the test installation to try again. There I found:
Once I install ucode-intel 20180312-3.1, the system slows down from 900% to 300% in the benchmark. Rolling back before that ucode package was installed, it gets fast again.
Could this explain why my laptops are not affected, while my three PCs are all hit by this slowdown? It is hard to believe that my 2012 laptop with a Core i5-3320M now visibly outperforms my 2017 PC with a i7-7700K. Let alone that the dually installed Windows 10 on the same PC does not suffer from any slowdown at all (using the Windows build of MAME).
That might be the Spectre V2 fix on new processors, which will slow it down. Can you check output of: cat /sys/devices/system/cpu/vulnerabilities/spectre_v2 Ciao, Marcus -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
That might be the Spectre V2 fix on new processors, which will slow it down. Can you check output of:
cat /sys/devices/system/cpu/vulnerabilities/spectre_v2
The latest postings did not reach the mailing list but only you. Thus, in order to repeat for the others: $ cat /sys/devices/system/cpu/vulnerabilities/spectre_v2 Mitigation: Indirect Branch Restricted Speculation, IBPB, IBRS_FW I could now verify on all my PCs (two i7-7xxx (Kaby Lake), one i7-6xxx (Skylake)) that when I add the boot parameter "spectre_v2=off", all machines return to their original fast performance. With this parameter, "Vulnerable" is reported for the above command. I reported this to the MAME forum, but some people had doubts that this is an expected effect. When MAME runs in benchmark mode, it does not produce screen or audio output. Actually, it should almost only execute in userland. The mitigation should affect system calls, not a user process. It looks as if the predictor gets stuck in the "off" position after the first syscall, one of them wrote. Greetings, Michael -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (5)
-
jdd＠dodin.org -
L A Walsh -
Marcus Meissner -
Michael Zapf -
Michal Kubecek