https://bugzilla.novell.com/show_bug.cgi?id=337003#c2 --- Comment #2 from Bernard Delley 2007-11-02 10:26:48 MST --- Created an attachment (id=181832) --> (https://bugzilla.novell.com/attachment.cgi?id=181832) messages while network connection was lost this is /var/log/messages from an event the made the machine loose network connection about Nov 2 9:45 am, it was not actually crashed. Connection was fixed by invoking yast and changing the inp address forth and back and the proceed to finish. and quit yast. this was in late afternoon about 16:48. The machine was running all the time on 4 CPU a mpich2 4-processor job with local mpd. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003#c4 --- Comment #4 from Bernard Delley 2007-11-02 10:35:19 MST --- Created an attachment (id=181836) --> (https://bugzilla.novell.com/attachment.cgi?id=181836) dmesg output just for further documentation -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 Mark Gordon changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181834|application/octet-stream |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 Mark Gordon changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.provo.novell.com |kernel-maintainers@forge.provo.novell.com Status|NEEDINFO |NEW Component|Basesystem |Kernel Info Provider|bernard.delley@psi.ch | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003#c6 --- Comment #6 from Nick Piggin 2007-11-07 19:19:03 MST --- The corrupted pte message means the pte unexpectedly points outside the range of the regular mem_map. There is not too much I can gather from the crash: The calltrace is pretty jumbled, but even if it was accurate, the it is only the path via which the corruption is detected, not where it occurred. vm_flags is VM_READ|VM_WRITE|VM_EXEC|VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC|VM_ACCOUNT It could be a library or program text mapping, I suppose (we don't print out vm_file or vm_ops information in these messages, so it is hard to know). The pte is (_PAGE_PRESENT|_PAGE_RW|_PAGE_USER|_PAGE_ACCESSED|_PAGE_DIRTY), which is not surprising -- just a regular user page. The corruption is likely to be in the higher order bits of the pte. It's almost impossible to make any useful progress on this unless you we can reproduce it. If you can reliably reproduce, we could eventually narrow it down. Although I do think it might be an overheating issue: it is very common to see the kernel hit memory corruption in pages and pagetables... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

From fn19.site (192.168.40.59) icmp_seq=1 Destination Host Unreachable Nov 8 14:03:10 fn18 mpd: fn18_38610 (runmainloop 308): no pulse_ack from rhs; re-entering ring

https://bugzilla.novell.com/show_bug.cgi?id=337003#c8 Bernard Delley changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181832|0 |1 is obsolete| | --- Comment #8 from Bernard Delley 2007-11-08 06:50:15 MST --- Created an attachment (id=182612) --> (https://bugzilla.novell.com/attachment.cgi?id=182612) var/log/messages while crashing the network interface on mpich2 slave I can reproducibly crash the network interface, by running a small mpi test program a few times. the program sends fake data between the two nodes. The machines remain almost idling. the program worked fine for example on older nodes operated with suse10.1 and on many other. here are the last line appearing in messages on fn18 after the mentioned actions were taken on fn19 or on fn18 console. ssh fn18 tail -f messages after reboot, mount, ssh root@fn18 Nov 8 13:56:25 fn18 sshd[3881]: Accepted publickey for root from 192.168.40.42 port 47757 ssh2 mpdboot -n 2 --verbose --chkup Nov 8 13:58:35 fn18 sshd[3928]: PAM audit_log_acct_message() failed: Operation not permitted mpdtrace -l Nov 8 13:59:08 fn18 sshd[3950]: PAM audit_log_acct_message() failed: Operation not permitted mpiexec -machinefile mpd.hosts -n 2 Mpt Nov 8 13:59:46 fn18 sshd[3979]: PAM audit_log_acct_message() failed: Operation not permitted ssh fn18 then exit Nov 8 14:00:26 fn18 sshd[4028]: Accepted publickey for delley from 192.168.40.59 port 45559 ssh2 Nov 8 14:00:29 fn18 sshd[4030]: PAM audit_log_acct_message() failed: Operation not permitted mpiexec -machinefile mpd.hosts -n 2 Mpt Nov 8 14:01:10 fn18 mpdman: mpdman starting new log; fn18_mpdman_1 sh fn18 then several times mpiexec -machinefile mpd.hosts -n 2 Mpt after a while: Nov 8 14:03:10 fn18 mpd: fn18_38610 (runmainloop 308): no pulse_ack from rhs; re-entering ring ping fn18 PING fn18.site (192.168.40.58) 56(84) bytes of data. the directly connected console showed messages at Nov 8 14:07:23 fn18 mpd: mpd ending mpdid=fn18_38610 (inside cleanup) at console /etc/init.d/network restart eth0 Nov 8 14:11:22 fn18 kernel: eth0: no IPv6 routers present then mpdboot etc after a few mpiexec network interfaces on fn18 and fn19 were dead and later reanimated from console with "network restart eth0" -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003#c10 --- Comment #10 from Nick Piggin 2007-11-08 18:50:38 MST --- Well I'm not sure, whether it is overheating or or other cause, but it does seem somewhat like hardware failure. On the other hand, the fact that your network driver is crashing indicates that the driver might have a bug in it. And the buggy driver could have corrupted memory and caused the 'Dmol39m90m message' (that message tells us the Dmol39m90m process had its page table memory corrupted). Let's put this problem aside for the moment and assume it is due to either a hardware or software bug elsewhere. It would be nice to try your workload with a different type of card / driver. I guess the realtek ethernet card is only a 100Mbps, so it won't help too much? Can you try it anyway? Or can you put a different (not Intel based) gigabit card in the system easily? The other thing we can try is to use a different version of the e1000 driver. I'm not an expert on the Linux network stack, so I don't konw the best way to go with this. Do you have the time, and are you comfortable with compiling and installing a new kernel? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003#c12 Nick Piggin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kkeil@novell.com --- Comment #12 from Nick Piggin 2007-11-14 17:41:29 MST --- OK, that should help a lot, and I agree it does strongly suggest a network driver (or possibly controller) problem. I hope it is also enough for your production usage so you can bear with us until we work out what the problem is... I believe the e1000 driver is the correct one. If it actually detects your card and brings up an interface, that suggests it matches the hardware PCI IDs... and given that Intel maintains both the hardware and the software, they should have this right. I'm not completely sure how the e1000 driver is maintained in the suse kernels, so I'm cc'ing Karsten. Thanks for your help and patience so far. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003#c14 --- Comment #14 from Bernard Delley 2007-11-16 02:35:59 MST --- Created an attachment (id=183654) --> (https://bugzilla.novell.com/attachment.cgi?id=183654) lspci out for identifying network chips I replaced the rtl8139 by an intel pro1000GT (PWLA8391GTBLK) pci card after setup by yast (+reboot) this eth2 works reliably (1h experience) this older intel Gb/s chip does 630Mb/s which is about half of the expected for more current onboard chips. trying e1000e for the onboard chip. rmmod e1000 modprobe e1000e yast knows neither of the ethernet controllers, so modprobe e1000 yast, specify hardware e1000e rmmod e1000 double check with lsmod : no e1000 is listed, just e1000e this gives ethernet connection. But, its unreliable again, maybe slightly better than with e1000 for now, the system is used with the intel pci card. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c16 --- Comment #16 from Bernard Delley 2007-12-19 06:47:04 MST --- same problem was found with Intel Q35 chipset, again ICH9 family: Intel DQ35joe mainboard core2 quad core processor Q6600 installation openSUSE-10.3-GM-DVD-x86_64.iso with tricks relating to acpi=on and dual or quad processors see report 328471 accidentally a boot with acpi=off was done, as reported in 328471 this make only one of the four cpu available -- not as intended with such a machine. BUT ! with safe settings (acpi=on) and consequently 1 cpu per node the on board ethernet appears to run without problems. so I will have to fall back using a pci network card for now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 Jeff Mahoney changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183654|application/octet-stream |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User kkeil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c17 Karsten Keil changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |bernard.delley@psi.ch --- Comment #17 from Karsten Keil 2008-01-24 07:29:58 MST --- Can you please try if the new e1000 7.6.15 driver from sourceforge.net work stable with your onboard card ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User john.ronciak@intel.com added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c19 --- Comment #19 from John Ronciak 2008-01-29 11:07:01 MST --- Is this a pre-production board this hang is happening on? What type of traffic and load does test program do? What does the output from /proc/interrupts show during the testing? This isn't the original problem being worked in this BZ is it? This is something new right? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c21 --- Comment #21 from Bernard Delley 2008-01-30 04:05:39 MST --- Created an attachment (id=192291) --> (https://bugzilla.novell.com/attachment.cgi?id=192291) output of lspci -vv on DQ35JO board in hindsight, I am not assured that I ever had a true system crash as originally reported. The beowulf node is sure not reachable from the network when bug has hit, and originally I rebooted the node. these are current intel boards with ich9 southgate. I assume that these are full scale production boards from a most reputed maker. they are shown on http://www.intel.com/products/desktop/motherboard/index.htm as DG33TL and DQ35JO my next older board I mentioned which works fine is DG965SS also shown on that page, it features an ich8 southgate. The small fortran mpi test program used in comment 18 does a series of mpi allreduce operations and records the time used. I add ethtool and lspci -vv and dmesg outputs for that DQ35JO board -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c23 --- Comment #23 from Bernard Delley 2008-01-30 04:10:21 MST --- Created an attachment (id=192293) --> (https://bugzilla.novell.com/attachment.cgi?id=192293) dmesg on DQ35JO board -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c25 Bernard Delley changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #192292|0 |1 is obsolete| | --- Comment #25 from Bernard Delley 2008-02-06 07:29:55 MST --- Created an attachment (id=193425) --> (https://bugzilla.novell.com/attachment.cgi?id=193425) ethtool -S from DQ35JO board here is the correct file showing ethtool output. I think, the dmesg output shows mainly the result of crontab -l # DO NOT EDIT THIS FILE - edit the master and reinstall. # (/tmp/crontab.XXXXzZSMVv installed on Wed Dec 19 17:05:16 2007) # (Cron version V5.0 -- $Id: crontab.c,v 1.12 2004/01/23 18:56:42 vixie Exp $) 1,11,21,31,41,51 * * * * /etc/init.d/network restart eth0 > /dev/null 2>& 1 its not a cable - switch etc problem! which resets the network in 10 min intervals. This ugly clue saves me walking to the room every time the network has been lost. there appear to be 'normal' sequences where the network was reset without need. and others which have "nfs: server fn02 not responding, still trying" this is actually saying that the network was dead, and it could not get through to the nfs server (the server is OK). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c27 Bernard Delley changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #192293|0 |1 is obsolete| | --- Comment #27 from Bernard Delley 2008-02-07 02:00:38 MST --- Created an attachment (id=193555) --> (https://bugzilla.novell.com/attachment.cgi?id=193555) dmesg (since reboot) done after network crash on DQ35jo the cron job resetting eth0 was removed, and a reboot done on node fn19, one of the DQ35jo nodes which together with the DG33tl nodes show all such network problems with the onboard network chip. please note the boot options in dmesg, these are related to bugs 325995 and 328471 network testing was done until the network interface crashed. the messages appearing on the terminal window on the two machines in the test are shown in the next comment. the network driver is e1000-7.6.15 as suggested in a previous comment dmesg was recorded before the network restart. the network restart added these lines to dmesg: ADDRCONF(NETDEV_UP): eth0: link is not ready e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready eth0: no IPv6 routers present usb 7-1: USB disconnect, address 2 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User john.ronciak@intel.com added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c29 --- Comment #29 from John Ronciak 2008-02-07 15:10:56 MST --- So let us review what is going on here. 1) the link keeps dropping and being regained. Cause is unknown but this is the crux of the problem, at least I think so. 2) Link loss can be due to lots of reasons. Here are a few questions that have not been answered yet: q1) Does this happen on other systems or just this one? q2) Does this happen with other tests like nttcp or iperf? q3) have other cables and ports (both switch and NIC) show the same problem. Above it's stated that "it's not a cable - switch etc problem". How do you know this? Bad cables cause link loss all the time. So do bad ports. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c30 --- Comment #30 from Bernard Delley 2008-05-19 03:15:25 MST --- comment 29 missed the message from 27 + 28: the frequent link loss was completely artificial due to a cron job doing network restart (to bring the computer back to a reachable state automatically). 27 + 28 report the basic problem, with permanent network loss on mpich2 runs - when the cron network restart is removed. so the needed info on the basic problem is in 27 and 28. new info: the DG35jo board appears to run correctly with the on-board network when kernel 2.6.24-default is used with SUSE 10.3 the same fix does not work for the other board in question. the DG33tl board reports that the on-board network is started and immediately after that hangs showing the login prompt line (it is a non-graphical installation) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User bernard.delley@psi.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c32 --- Comment #32 from Bernard Delley 2008-05-20 00:58:06 MST --- NO ! its not a loose contact or the like. and we are talking about 4 machines two for each mainboard. which show the problems addressed here. DQ35jo 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connnecti works now with 2.6.24 kernel DG33tl when used with the onboard 00:19.0 Ethernet controller: Intel Corporation 82801I (ICH9 Family) Gigabit Ethern gets into hang state on use with SUSE 10.3 default kernel and hangs on boot with 2.6.24-default kernel. ==> this is the remaining problem of interest. on DG33tl I use for production calculations at reduced performance the additional 06:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05) this older pci plugin card works fine, but performance is about half of the on-board chip. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User digidietze@draisberghof.de added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c34 --- Comment #34 from Josua Dietze 2008-07-09 12:30:03 MDT --- Created an attachment (id=226828) --> (https://bugzilla.novell.com/attachment.cgi?id=226828) Oops of Fedora PAE kernel when uploading microcode -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User kkeil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c36 Karsten Keil changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |bernard.delley@psi.ch --- Comment #36 from Karsten Keil 2008-08-07 09:59:32 MDT --- So a plain 2.6.24 works stable with DQ35joe board but doesn't boot on DG33TL because a kernel crash at boot ? Do you have details about the crash ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User kkeil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c37 --- Comment #37 from Karsten Keil 2008-10-24 08:42:13 MDT --- If it is the microcode crash from the attachment, what if you boot without loading the microcode module ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=337003 User kkeil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=337003#c38 Karsten Keil changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |RESOLVED Info Provider|bernard.delley@psi.ch | Resolution| |NORESPONSE --- Comment #38 from Karsten Keil 2009-02-06 06:44:36 MST --- OK no answer, so I close this now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.