https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c2 Adolfo Salazar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|aosalazari@gmail.com | --- Comment #2 from Adolfo Salazar 2014-05-26 07:56:04 UTC --- Created an attachment (id=591954) --> (http://bugzilla.novell.com/attachment.cgi?id=591954) The output of hwinfo -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c4 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |aosalazari@gmail.com --- Comment #4 from Takashi Iwai 2014-05-26 08:36:34 UTC --- OK, thanks. One more favor: could you give the kernel messages (the output of dmesg) at booting both 3.11.10-7 and 3.11.10-11 kernels? I took a quick look though the commits between them, but no obvious culprit is found. We may need to compare the boot log in such a case. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c6 --- Comment #6 from Adolfo Salazar 2014-05-26 10:45:41 UTC --- Created an attachment (id=592000) --> (http://bugzilla.novell.com/attachment.cgi?id=592000) dmesg log files for kernels 3.11.10-7 and 3.11.10-11 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c8 --- Comment #8 from Takashi Iwai 2014-05-26 11:30:38 UTC --- (In reply to comment #7)

results of the alternative methods for hibernating:

1. echo disk > /sys/power/state No messages displayed on console, upon restarting the grub menu is shown, restores to last state

2. powersave -U Messages are displayed on console, no grub menu or login prompt, restores last state

3. pm-hibernate behaves the same as powersave -U

So, all these three manual methods did resume after S4 successfully?

Additionally, I enabled kdump and that causes hibernate to work as expected. Disabling kdump reverts to the buggy state.

Do you mean S4 gets broken with the methods above once when kdump is disabled? Or, is it only about S4 from KDE menu? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c10 Adolfo Salazar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|aosalazari@gmail.com | --- Comment #10 from Adolfo Salazar 2014-05-26 23:45:52 UTC --- Here are some additional results: 1. I had done a shutdown and rebooted the PC. The system was working as expected and I had firefox and okular running. I then opened a terminal window and ran the "powersave -U" command. This time the system locked up as per my previous files. I then rebooted the pc and using the terminal window ran "echo disk > /sys/power/state" command and the system shutdown as in my earlier post. Upon rebooting I once again ran the "powersave -U" command and this time it hibernated as per my earlier reports. At a pinch I say that the key to get the powersave command to work is that I must run the "echo disk" command first. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c12 --- Comment #12 from Adolfo Salazar 2014-05-27 22:53:02 UTC --- I have re-run the tests with the addition of using the command. the results are the same and the results of <uname> are: [quote] linux-ecrt-aos:/home/adolfo # uname -a Linux linux-ecrt-aos 3.11.10-11-desktop #1 SMP PREEMPT Mon May 12 13:37:06 UTC 2014 (3d22b5f) i686 i686 i386 GNU/Linux linux-ecrt-aos:/home/adolfo # linux-ecrt-aos:/home/adolfo # uname -a Linux linux-ecrt-aos 3.11.10-7-desktop #1 SMP PREEMPT Mon Feb 3 09:41:24 UTC 2014 (750023e) i686 i686 i386 GNU/Linux linux-ecrt-aos:/home/adolfo # [/quote] Are there any other tests I can do to ensure that there was a regression of kernel? BTW how can I submit multiple file attachments? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c14 --- Comment #14 from Takashi Iwai 2014-05-28 13:00:36 UTC --- (In reply to comment #12)

I have re-run the tests with the addition of using the command. the results are the same

Do you mean that 3.11.10-7 worked and 3.11.10-11 was broken?

and the results of <uname> are:

[quote] linux-ecrt-aos:/home/adolfo # uname -a Linux linux-ecrt-aos 3.11.10-11-desktop #1 SMP PREEMPT Mon May 12 13:37:06 UTC 2014 (3d22b5f) i686 i686 i386 GNU/Linux linux-ecrt-aos:/home/adolfo #

linux-ecrt-aos:/home/adolfo # uname -a Linux linux-ecrt-aos 3.11.10-7-desktop #1 SMP PREEMPT Mon Feb 3 09:41:24 UTC 2014 (750023e) i686 i686 i386 GNU/Linux linux-ecrt-aos:/home/adolfo # [/quote]

Are there any other tests I can do to ensure that there was a regression of kernel?

Nothing can ensure 100%. But if booting with one kernel always works and another always fails for a few times tests, it's mostly sure.

BTW how can I submit multiple file attachments?

Use tar or zip. Or just attach multiple times. Now how to proceed. Since it's apparently a regression between two 3.11.x.y kernels, and the bug is easily reproducible, the best would be to find out the regression by git bisect. If you need to know how to build a kernel and perform git bisect, I can give you a brief instruction. The expanded git tree is available at git@gitorious.org:opensuse/kernel.git and the bisection range is b73e414b..1cb3e62f. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c16 --- Comment #16 from Takashi Iwai 2014-05-29 07:48:39 UTC --- (In reply to comment #15)

...

Now how to proceed. Since it's apparently a regression between two 3.11.x.y kernels, and the bug is easily reproducible, the best would be to find out the regression by git bisect.

If you need to know how to build a kernel and perform git bisect, I can give you a brief instruction. The expanded git tree is available at git@gitorious.org:opensuse/kernel.git and the bisection range is b73e414b..1cb3e62f.

I have never used git bisect before so any pointers will be useful.

The first thing to do is to build your kernel from git tree. 1. Clone the git tree. Suppose you'll expand the tree under $HOME/git % mkdir $HOME/git % cd $HOME/git % git clone git@gitorious.org:opensuse/kernel.git 2. Switch to openSUSE-13.1 branch % cd kernel % git checkout openSUSE-13.1 3. Go back to the good working commit % git reset --hard b73e414b 4. Set up configuration; below will give you the minimal config that is similar to the running kernel. % make localmodconfig 5. Edit .config file; it'd be better to change CONFIG_LOCALVERSION to a unique one like "-test". Also, better to backup .config file as a different name at this stage. 6. Run make; you can build parallel by -j 4 or so (4 is the number of CPUs) 7. If everything is OK, you'll have a kernel in arch/x86/boot/bzImage. Run "make install" as root. "make modules_install" will install only the modules. Run "mkinitrd" if necessary, too. 8. Reboot with your kernel. If it doesn't appear in GRUB menu, check the grub configuration. There is a helper script /sbin/update-bootloader. Pass --image and --initrd with the new vmlinuz and initrd paths. 9. Test S4. With this kernel, S4 must work. 10. After confirming the successful S4, try the newer kernel. % git reset --hard 1cb3e62f Then build again. You can just run "make -j4". 11. Install and test this kernel (you can override the old one). Confirm that S4 is broken with this. OK, now we can start bisection. 12. Run the following: % git bisect start % git bisect bad 1cb3e62f % git bisect good b73e414b This will checkout the commit to test automatically. Build this, install and test the kernel. 13. If S4 is still broken with the current kernel, give "git bisect bad". OTOH, if S4 works with the current kernel, give "git bisect good". Then git will give you the next commit to test. Repeat this until you reach to the regression commit. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c18 --- Comment #18 from Adolfo Salazar 2014-05-30 02:00:28 UTC --- Please ignore the above. I thought that I had initialised the RSA keys. Right now it is compiling away. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c20 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mhocko@suse.com --- Comment #20 from Takashi Iwai 2014-05-30 11:09:43 UTC --- Great, then could you try to revert the commit? - Reset git bisect % git bisect reset - Go to the latest kernel state % git reset --hard origin/openSUSE-13.1 - Revert the commit % git revert d4a930379086bdda70672821bb6674a2e26a8e13 - Build the kernel again and check whether the problem is gone. Meanwhile, I add Michal to Cc for taking a deeper look at it. For removing the kernel entry, # update-bootloader --remove --image /boot/vmlinuz-$YOURS \ --initrd /boot/initrd-$YOURS # rm /boot/vmlinuz-$YOURS # rm /boot/initrd-$YOURS # rm /boot/System.map-$YOURS # rm -r /lib/modules/$YOURS -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c22 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|kernel-maintainers@forge.pr |mhocko@suse.com |ovo.novell.com | --- Comment #22 from Takashi Iwai 2014-05-31 07:48:18 UTC --- Thanks. Michal, the patch seems leading to a regression on user-space S4. I reassign this bug to you to decide whether to more debug or to just revert it for openSUSE-13.1. On the newer distros including SLE12, this problem would hit rarely, I guess, because user-space S4 is no longer invoked there. Instead, systemd provides only kernel-space S4. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c24 --- Comment #24 from Adolfo Salazar 2014-06-03 01:18:40 UTC --- The only thing that I have running when I have been testing for hibernation is the KDE plasma and, sometimes terminal console and Mozilla displaying the opensuse homepage. For the test you asked me to conduct I only ran the terminal console and no Mozilla to capture the data you requested. I had difficulty getting the snapshots as these files read that they were zero bytes long. Here is the out put as requested: Linux linux-ecrt-aos 3.11.10-11-desktop #1 SMP PREEMPT Mon May 12 13:37:06 UTC 2014 (3d22b5f) i686 i686 i386 GNU/Linux /proc/sys/vm/dirty_background_bytes:0 /proc/sys/vm/dirty_background_ratio:10 /proc/sys/vm/dirty_bytes:0 /proc/sys/vm/dirty_expire_centisecs:3000 /proc/sys/vm/dirty_ratio:20 /proc/sys/vm/dirty_writeback_centisecs:500 -r--r--r-- 1 root root 0 Jun 3 10:59 meminfo -r--r--r-- 1 root root 0 Jun 3 11:00 vmstat I hope that this is what you were looking for. If not let me know what steps to take to get the info you need. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c26 --- Comment #26 from Adolfo Salazar 2014-06-04 04:43:15 UTC --- Created an attachment (id=593213) --> (http://bugzilla.novell.com/attachment.cgi?id=593213) Log files as requested I have attached the meminfo and vmstat files as per your instructions. Please note that I normally use the KDE desktop so when I use the hibernate mode it is from the "Application Launcher -> Leave -> Hibernate". I have also had teh same result from a terminal console logged in as root. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c28 --- Comment #28 from Michal Hocko 2014-06-05 14:27:26 CEST --- Created an attachment (id=593483) --> (http://bugzilla.novell.com/attachment.cgi?id=593483) patch to try The patch applies cleanly and it might help out here. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c30 --- Comment #30 from Michal Hocko 2014-06-05 15:27:58 CEST --- Great! Thanks Takashi for testing. I will push a1c3bfb2 into 13.1 and other branches where it is applicable. The patch it reverts looks really bogus and random. I am quite surprised it made it in. (In reply to comment #29) [...]

So, removing the redundant min_free_kbytes by the upstream patch you suggested would fix the problem, but it's still fairly tight in comparison with the former kernels. Before the commit [a1c3bfb2: mm/page-writeback.c: do not count anon pages as dirtyable memory], it took anon pages into account. And there are about 80000 anon pages available at the time of image preallocation.

That is true but adding anon pages to the picture just papers over the inherent problem here. There doesn't make any sense to throttle the hibernation process. This is handled with the in-kernel hibernation because the kernel thread is most probably marked to be not throttled (I haven't checked but that would make a lot of sense). Anyway, let's wait for Adolfo to confirm your observations. Adolfo, could you test with the attached patch, please? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c32 --- Comment #32 from Michal Hocko 2014-06-06 10:03:18 CEST --- (In reply to comment #31)

I have tested the system 3 times with the patched kernel and hibernate works as expected.

Great!

Micha I would ask one favour from you - it seems that you have overestimated my capabilities a little bit. I haven't coded for about 14 years and that was with specialist embedded systems, so my knowledge of linux and relevant tools are quite basic. So please can you give me detailed instructions as to what you need me to do so at least I feel a little bit secure that I am doing things the way you want me to do them. Having said that I had forgotten how much fun this is. let me know what you need me to do next.

Well, your part is done. Keep running your patched kernel until a new maintenance update is released. It will contain the fix + some more which will be part of that update as well.

PS I did learn how to incorporate the patch using git so I am very confident that the process is fine.

-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Status Whiteboard| |obs:running:2876:important -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=879792 https://bugzilla.novell.com/show_bug.cgi?id=879792#c35 --- Comment #35 from Swamp Workflow Management 2014-06-25 07:09:16 UTC --- openSUSE-SU-2014:0840-1: An update that solves 9 vulnerabilities and has 15 fixes is now available. Category: security (important) Bug References: 851338,858067,868315,869563,870173,870576,871561,872715,873374,876102,876981,877257,877713,877721,878115,878274,879258,879792,880599,880613,880892,881697,881727,882648 CVE References: CVE-2013-7339,CVE-2014-0055,CVE-2014-0077,CVE-2014-2678,CVE-2014-2851,CVE-2014-3122,CVE-2014-3144,CVE-2014-3145,CVE-2014-3153 Sources used: openSUSE 13.1 (src): cloop-2.639-11.10.1, crash-7.0.2-2.10.9, hdjmod-1.28-16.10.1, ipset-6.21.1-2.14.1, iscsitarget-1.4.20.3-13.10.1, kernel-docs-3.11.10-17.6, kernel-source-3.11.10-17.2, kernel-syms-3.11.10-17.1, ndiswrapper-1.58-10.1, pcfclock-0.44-258.10.1, vhba-kmp-20130607-2.11.1, virtualbox-4.2.18-2.15.2, xen-4.3.2_01-18.2, xtables-addons-2.3-2.10.1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.