https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c2 Andrew Holland changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew.holland@nordicteleco | |m.fi --- Comment #2 from Andrew Holland 2013-06-24 09:19:10 UTC --- I can't get a photo, for several reasons: #1. There is no physical display, so everything is over a serial console or network. #2. These spit out far too quickly to capture via a photograph. #3. The hardware in question has needed to be used for production, so unfortunately I can't test on that machine without affecting production. If you can work with a serial console log of the session, I can grab it during a downtime that I will schedule. That said, the information you'll get from it is all in the report above - you'll just have thousands upon thousands of lines of these: Jun 07 11:25:38 mysql1.nordictele.com kernel: sdc1: rw=129, want=8384512, limit=1951744 Jun 07 11:25:38 mysql1.nordictele.com systemd-journal[379]: Missed 240 kernel messages And a few of these, on a seperate tty: kernel:[ 148.953735] BUG: soft lockup - CPU#5 stuck for 22s! [mkfs.ext4:4631] They'll have different time offsets, of course. If the thousands of repeated items help, I'll schedule the maintenance window and grab the console log. Would you need any other info gathered at that time, or would you like any other tests run during the maintenance window? -- actually, systemd's journal would still probably have the logs and I can grab those instead. I will check and if I can retrieve them, I'll attach in a reply. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c4 Andrew Holland changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|andrew.holland@nordicteleco | |m.fi | --- Comment #4 from Andrew Holland 2013-06-24 10:20:05 UTC --- Created an attachment (id=545365) --> (http://bugzilla.novell.com/attachment.cgi?id=545365) xz compressed dmesg/systemd log of session Dmesg currently doesn't show the initial boot information - so here is a log that includes both dmesg and the entire journalctl stored info (standard journalctl options other than --no-pager ) File is compressed with xz -9, as it is somewhat large and contains a lot of repeated info; xzcat [file] | less or xz -dc [file] | less is probably the most effective way to view. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c6 --- Comment #6 from Jiri Slaby 2013-06-24 12:52:23 UTC --- Created an attachment (id=545374) --> (http://bugzilla.novell.com/attachment.cgi?id=545374) dmesg in plaintext -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c8 Andrew Holland changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|andrew.holland@nordicteleco | |m.fi | --- Comment #8 from Andrew Holland 2013-06-25 03:21:50 UTC --- Interestingly, discard_max_bytes does not exist for /sys/block/sdc: mysql1:/home/andrew # cd /sys/block/sdc mysql1:/sys/block/sdc # du 0 ./sdc1/power 0 ./sdc1/trace 0 ./sdc1/holders 0 ./sdc1 0 ./power 0 ./queue/iosched 0 ./queue 0 ./trace 0 ./slaves 0 ./holders 0 . mysql1:/sys/block/sdc # cd queue mysql1:/sys/block/sdc/queue # ls -la total 0 drwxr-xr-x 3 root root 0 Jun 25 14:11 . drwxr-xr-x 8 root root 0 Jun 13 14:56 .. -rw-r--r-- 1 root root 4096 Jun 25 14:11 add_random -r--r--r-- 1 root root 4096 Jun 25 14:11 discard_granularity -r--r--r-- 1 root root 4096 Jun 25 14:11 discard_max_bytes -r--r--r-- 1 root root 4096 Jun 25 14:11 discard_zeroes_data -r--r--r-- 1 root root 4096 Jun 25 14:11 hw_sector_size drwxr-xr-x 2 root root 0 Jun 25 14:11 iosched -rw-r--r-- 1 root root 4096 Jun 25 14:11 iostats -r--r--r-- 1 root root 4096 Jun 25 14:11 logical_block_size -r--r--r-- 1 root root 4096 Jun 25 14:11 max_hw_sectors_kb -r--r--r-- 1 root root 4096 Jun 25 14:11 max_integrity_segments -rw-r--r-- 1 root root 4096 Jun 25 14:11 max_sectors_kb -r--r--r-- 1 root root 4096 Jun 25 14:11 max_segment_size -r--r--r-- 1 root root 4096 Jun 25 14:11 max_segments -r--r--r-- 1 root root 4096 Jun 25 14:11 minimum_io_size -rw-r--r-- 1 root root 4096 Jun 25 14:11 nomerges -rw-r--r-- 1 root root 4096 Jun 25 14:11 nr_requests -r--r--r-- 1 root root 4096 Jun 25 14:11 optimal_io_size -r--r--r-- 1 root root 4096 Jun 25 14:11 physical_block_size -rw-r--r-- 1 root root 4096 Jun 25 14:11 read_ahead_kb -rw-r--r-- 1 root root 4096 Jun 25 14:11 rotational -rw-r--r-- 1 root root 4096 Jun 25 14:11 rq_affinity -rw-r--r-- 1 root root 4096 Jun 25 14:11 scheduler -r--r--r-- 1 root root 4096 Jun 25 14:11 write_same_max_bytes - so there is no "discard_max_bytes" The remainder are: sdc/queue/discard_max_bytes: 15728640 sdc/discard_alignment: 0 sdc1/discard_alignment: 14680064 sdc1/alignment_offset: 0 --- mysql1:/sys/block/sdc/queue # cat discard_granularity 15728640 mysql1:/sys/block/sdc/queue # cd .. mysql1:/sys/block/sdc # cat discard_alignment 0 mysql1:/sys/block/sdc/sdc1 # cat discard_alignment 14680064 mysql1:/sys/block/sdc/sdc1 # cat alignment_offset 0 --- -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c10 Jan Kara changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|kernel-maintainers@forge.pr |jack@suse.com |ovo.novell.com | --- Comment #10 from Jan Kara 2013-06-25 03:49:55 UTC --- Yeah, ok, the values explain the problem. 'discard_granularity' isn't a power of two so the logic and masking in blkdev_issue_discard() starts giving random numbers and eventually tries to trim negative number of blocks which has fancy results. Looking into the upstream code, the problem seems to be fixed there by commit 8dd2cb7e880d2f77fba53b523c99133ad5054cfd. I'll port the patch to our kernel. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c12 --- Comment #12 from Andrew Holland 2013-06-25 04:07:14 UTC --- I can test starting at Midnight Helsinki time - would that be too early? (What time zone is used for builds - CET? UTC? MDT?) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c14 --- Comment #14 from Andrew Holland 2013-06-26 02:43:40 UTC --- I'm sorry, I just cannot find an rpm package that is not built from the 13th of this month in that location, or on the kernel.opensuse.org site, for x86_64. The kernel from the 13th still has the issue (as expected.) I see the patch, and I may just compile it myself to test. I'll keep looking for a package in the meanwhile. Aha - I see one at https://build.opensuse.org/package/binary?arch=x86_64&filename=kernel-default-3.7.10-34.1.g0ce5047.x86_64.rpm&package=kernel-default&project=Kernel%3AopenSUSE-12.3&repository=standard built an hour ago. I shall attempt to test that package. (I've repurposed the machine for testing today.) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c16 --- Comment #16 from Jiri Slaby 2013-06-26 03:04:08 UTC --- It's not built yet... https://build.opensuse.org/project/monitor?project=Kernel%3AopenSUSE-12.3 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c18 Andrew Holland changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|andrew.holland@nordicteleco | |m.fi | --- Comment #18 from Andrew Holland 2013-06-26 05:04:26 UTC --- I am happy to say this patch has solved the issue. (As did the workaround, but I didn't want to leave this in a state where if I went on vacation or similar, someone else might bring the box to it's knees by using a standard mkfs command.) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=823797 https://bugzilla.novell.com/show_bug.cgi?id=823797#c Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Status Whiteboard| |obs:running:2034:moderate -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=823797 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Whiteboard|obs:running:2034:moderate | -- You are receiving this mail because: You are on the CC list for the bug.