https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c2 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO CC| |jslaby@novell.com InfoProvider| |drichard@largo.com --- Comment #2 from Jiri Slaby 2011-08-20 07:47:24 UTC --- Just a shot in the dark. Does latencytop show anything useful? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c4 Michal Hocko changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mhocko@novell.com --- Comment #4 from Michal Hocko 2011-08-22 16:23:56 CEST --- (In reply to comment #0) [...]

The first visible sign of problem is the network. Here is the network stats on an OpenSuse 11.3 server which is being hammered with networking and working well. Note the low RX packets.

eth0 Link encap:Ethernet HWaddr F4:CE:46:C0:EA:A8 inet addr:128.222.233.237 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1970250698 errors:0 dropped:175 overruns:0 frame:0

On this OpenSuse 11.4 server (which is used to deploy GNOME to thin clients), here is the same information:

eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72 inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:67783895 errors:0 dropped:175438 overruns:0 frame:0

You seem to have _much_ bigger packet drop with 11.4 wrt 11.3. Is any of your infrastructure relying on the network (ldap...)

The other very visible issue is disk performance. I've noticed: - If you type in 'vi /etc/hosts' it sits for 3-4 seconds with a blinking cursor before the file appears. I tried it from the console and it's doing the same thing.

Have you tried to strace/ltrace this command with timing information? (strace -tt -o strace.log vi /etc/hosts)

- If you change a password for a user, it sits for 3-4 seconds before the prompt comes back.

This would point to ldap. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c6 --- Comment #6 from David Richards 2011-08-22 17:07:14 UTC --- @all: I can provide a ssh tunnel to this server if anyone wants to see it first hand. These conditions (40 people working concurrently) would be impossible to reproduce on a test machine. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c8 David Richards changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|drichard@largo.com | --- Comment #8 from David Richards 2011-08-22 19:27:51 UTC --- @Greg: Attaching hwinfo. The other servers are running similar hardware. Here is some more information: We believe the networking problem is in the operating system because we duped the physical machine and moved it to VMWare. The VM server is using Intel cards vs the physical server which is running Broadcom. On the VM: eth0 Link encap:Ethernet HWaddr 00:50:56:8B:00:05 inet addr:128.222.99.242 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:24185635 errors:0 dropped:548189 overruns:0 frame:0 So it's doing the same thing as the physical machine, dropping tons of packets. Back to the disk IO issue, I found something very odd. I'm seeing speed differences based on the sub-directory, even though we only have one partition of / ! If I vi /etc/hosts it takes about 4-5 seconds to open. I then copied the hosts file into various folders. If I vi /tmp/hosts it opens immediately. If I vi /home/hosts, it opens in about 2 seconds. I then placed it on an NFS mount and that takes about 2 seconds to open. This is just crazy: /dev/disk/by-id/cciss-3600508b100184a395358363055330002-part1 swap swap defaults 0 0 /dev/disk/by-id/cciss-3600508b100184a395358363055330002-part2 / ext4 acl,user_xattr,barrier=0 1 1 /etc, /tmp and /home are on the same physical mount. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c10 Greg Kroah-Hartman changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |drichard@largo.com --- Comment #10 from Greg Kroah-Hartman 2011-08-22 19:56:31 UTC --- (In reply to comment #9)

Created an attachment (id=447027) --> (http://bugzilla.novell.com/attachment.cgi?id=447027) [details] Output from hwinfo

Are you sure? It usually starts out looking like: ============ start debug info ============ libhd version 18.3u (x86-64) using /var/lib/hardware .. and then ends with the pci information you provided. Care to try attaching the whole thing? Perhaps it needs to be compressed to let bugzilla accept it? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c12 David Richards changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|drichard@largo.com | --- Comment #12 from David Richards 2011-08-22 20:14:39 UTC --- We are going to turn off AppArmor tonight at 5pm and see if that affects performance. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c14 --- Comment #14 from David Richards 2011-08-23 13:51:05 UTC --- @greg: AppArmor was unloaded last night and made no change to the performance. I had to Google IPMI, so I guess we aren't using it. :) I modprobe -r'd the module and dmesg reports the module unloaded. No change. We have 41 users logged in today and it's taking almost 5 seconds for /etc/hosts to open with vi. I mentioned yesterday that when I cp /etc/hosts to /tmp/hosts and then vi that file it opens immediately. If I cp /etc/hosts to /etc/hosts2 and then vi /etc/hosts2 it's still slow. It's the oddest thing I have ever seen. When we get 2 users in Nautilus, the server just comes to a crawl and freezes. There is definitely something wrong with disk access. We have many servers with two NICs and have never seen this problem. Our older GNOME servers are configured in the same manner. We have Evolution running on SLED and it's configured with two networks too. Network RX errors continue to grow. When I check about every 1-2 seconds it usually goes up at least by one, sometimes more. eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72 inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:141457074 errors:0 dropped:537878 overruns:0 frame:0 We will download, install and boot the 3.0 kernel. I'll try and schedule that tonight when the users are off. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c16 --- Comment #16 from David Richards 2011-08-23 14:13:27 UTC --- @all: I appreciate all of your expertise in this area, I don't open bugs until we are stumped. Attaching the strace after this message. vi of /etc/services is slow too. It's just crazy that certain areas of the disk are hammered and others not, when they are all mounted in / ! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c18 --- Comment #18 from Michal Hocko 2011-08-23 16:27:09 CEST --- (In reply to comment #17)

Created an attachment (id=447195) --> (http://bugzilla.novell.com/attachment.cgi?id=447195) [details] strace of opening /etc/hosts with vi

Could you run strace with -tt -f parameters so we get the timing information and catch also all forked processes, please? Could you also give us /proc/mounts? Could you also run vmstat 1 and take the output while you are accessing the file that shows an issue. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c David Richards changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #447199|application/octet-stream |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c21 --- Comment #21 from David Richards 2011-08-23 14:49:14 UTC --- I started vmstat and it's seen when cpu/id got to 23. When it returns to 2, that's when I :q'd out of vi. vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 4 0 0 48967620 257616 9364624 0 0 0 2 3 3 1 1 98 0 0 0 0 0 48967356 257616 9364628 0 0 0 112 2884 1355 1 7 92 0 0 0 0 0 48967356 257616 9364624 0 0 0 0 1936 1083 0 5 94 0 0 3 0 0 48966356 257616 9364624 0 0 0 0 2873 1472 1 9 90 0 0 9 0 0 48965852 257616 9364624 0 0 0 0 4480 2233 1 23 76 0 0 3 0 0 48965852 257616 9364624 0 0 0 0 4415 2226 1 24 75 0 0 1 0 0 48965836 257616 9364624 0 0 0 0 5186 2205 1 26 73 0 0 1 0 0 48966472 257616 9364624 0 0 0 88 4186 1974 1 19 79 0 0 0 0 0 48965860 257616 9364624 0 0 0 344 1128 516 0 2 98 0 0 1 0 0 48965860 257616 9364624 0 0 0 0 2272 1064 1 5 94 0 0 1 0 0 48964436 257616 9364624 0 0 4 4 6569 1971 1 28 70 0 0 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c23 --- Comment #23 from David Richards 2011-08-23 16:11:52 UTC --- Created an attachment (id=447224) --> (http://bugzilla.novell.com/attachment.cgi?id=447224) Showing cpus/hypercores. Attaching all information that I can think of, this server is 4 CPU x 4 core. Top rarely/never shows more than a few percentage busy. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c25 --- Comment #25 from David Richards 2011-08-23 16:31:30 UTC --- @Jiri: I had to google sysrq-t and think I did what you are asking. I echo'd a t to /proc/sysrq-trigger and it threw lines into /var/log/messages. Attaching the output, bzip'd -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c27 --- Comment #27 from David Richards 2011-08-23 16:45:52 UTC --- @all: Also of interest, comparing the old GNOME server with the new one reveals that many more files are now open. I'll attach the lsof output from OpenSuse 11.4. OpenSuse 10.2, 140 users logged in lsof | wc -l 240444 OpenSuse 11.4, 40 users logged in lsof | wc -l 287837 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c29 --- Comment #29 from Jiri Slaby 2011-08-23 17:12:14 UTC --- (In reply to comment #24)

Created an attachment (id=447225) --> (http://bugzilla.novell.com/attachment.cgi?id=447225) [details] strace -o /tmp/vietchostsmore.out -tt -f -T vi /etc/hosts

Interesting times (100x slower than on my machine): open("/etc/.hosts.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 <0.411674> .. open("/etc/.hosts.swpx", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 <0.339978> .. close(5) = 0 <0.461197> unlink("/etc/.hosts.swpx") = 0 <0.404862> close(4) = 0 <0.313377> unlink("/etc/.hosts.swp") = 0 <0.407398> .. open("/etc/.hosts.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 <0.445237> .. write(4, "..."..., 4096) = 4096 <0.417035>

sysrq-trigger was passed a "t" and this was dumped.

This is not useful much, it's full of scheduler stats. Could you run # dmesg -c -s 1000000 and then # echo t > /proc/sysrq-trigger # dmesg > log and attach log? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c31 --- Comment #31 from David Richards 2011-08-24 12:51:08 UTC --- @all: We did not install the 3.0 kernel last night because it seemed like we were getting good data out of the current one. Does everyone feel we should upgrade? I can limp along a few more days if this information is useful. Otherwise, I'll upgrade and reboot tomorrow. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c32 Michal Hocko changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |drichard@largo.com --- Comment #32 from Michal Hocko 2011-08-24 15:26:04 CEST --- (In reply to comment #30)

Created an attachment (id=447246) --> (http://bugzilla.novell.com/attachment.cgi?id=447246) [details] Output of dmesg after -c -s and then sysrg-trigger to t

This still doesn't contain anything useful (any traces): $ head bug-712958_dmesg.out 000000 [631329.878618] .se->statistics.exec_max : 1.528704 [631329.878618] .se->statistics.slice_max : 1.122945 [631329.878618] .se->statistics.wait_max : 2.041506 [631329.878618] .se->statistics.wait_sum : 3.526940 [631329.878618] .se->statistics.wait_count : 30 [631329.878618] .se->load.weight : 512 Anyway, as Jiri already pointed out. File open taking almost half a sec looks really suspicious. Let's have a look at the worst (successful) file open times: $ grep "\" bug-712958_vietchostsmore.log | grep -v "= -1" | sed 's/.*<\(.*\)>/\1/' | sort | tail -n5 0.339978 0.379258 0.403296 0.411674 0.445237 Which are those files? $ grep "\" bug-712958_vietchostsmore.log | grep -v "= -1" | sed 's/.*<\(.*\)>/\1/' | sort | tail -n5 > worst_open $ grep -f worst_open bug-712958_vietchostsmore.log 24197 12:13:07.959485 open("/etc/.hosts.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 <0.411674> 24197 12:13:08.382062 open("/etc/.hosts.swpx", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 <0.339978> 24197 12:13:10.370000 open("/etc/.hosts.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 <0.445237> 24197 12:13:14.776717 open("/etc/4913", O_WRONLY|O_CREAT|O_EXCL, 0100644) = 3 <0.379258> 24197 12:13:17.044206 open("/etc/hosts", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 3 <0.403296> Let's have a look at all file opens from /etc: $ grep "open.*etc" bug-712958_vietchostsmore.log 24197 12:13:07.042719 open("/etc/ld.so.cache", O_RDONLY) = 3 <0.002339> 24197 12:13:07.465060 open("/etc/vimrc", O_RDONLY) = 3 <0.000014> 24197 12:13:07.789405 open("/etc/nsswitch.conf", O_RDONLY) = 3 <0.000467> 24197 12:13:07.812897 open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000534> 24197 12:13:07.841753 open("/etc/ld.so.cache", O_RDONLY) = 3 <0.001435> 24197 12:13:07.907670 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3 <0.000288> 24197 12:13:07.949504 open("/etc/hosts", O_RDONLY) = 3 <0.000467> 24197 12:13:07.955735 open("/etc/.hosts.swp", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000703> 24197 12:13:07.959485 open("/etc/.hosts.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 <0.411674> 24197 12:13:08.378219 open("/etc/.hosts.swpx", O_RDONLY) = -1 ENOENT (No such file or directory) <0.001717> 24197 12:13:08.382062 open("/etc/.hosts.swpx", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 <0.339978> 24197 12:13:10.370000 open("/etc/.hosts.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 <0.445237> 24197 12:13:11.601865 open("/etc/hosts", O_RDONLY) = 3 <0.000019> 24197 12:13:14.776717 open("/etc/4913", O_WRONLY|O_CREAT|O_EXCL, 0100644) = 3 <0.379258> 24197 12:13:17.044206 open("/etc/hosts", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 3 <0.403296>

From this, it looks that any write access is terribly slow (O_RDWR or O_WRONLY). I guess that /etc/ is at your root partition which seems to be ext4? Have you formated the filesystem from scratch or migrated it from ext3? Is the filesystem almost full?

You said that if you copied the file to /tmp then it opens just fine. Is /tmp on the same root partition or is it a link to tmpfs which you have mounted? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c34 --- Comment #34 from Jiri Slaby 2011-08-24 15:45:40 UTC --- (In reply to comment #32)

(In reply to comment #30)

...

Created an attachment (id=447246) --> (http://bugzilla.novell.com/attachment.cgi?id=447246) [details] [details] Output of dmesg after -c -s and then sysrg-trigger to t

This still doesn't contain anything useful (any traces): $ head bug-712958_dmesg.out .000000 [631329.878618] .se->statistics.exec_max : 1.528704 [631329.878618] .se->statistics.slice_max : 1.122945

Why is this crap a part of sysrq-t output at all? The only option here is to increase the log buffer. It can be done by the log_buf_len kernel option. Something like log_buf_len=1000000. So if you are going to reboot, pass this in. Then, when you encounter this weird behaviour, repeat the process with dmesg+sysrq-trigger. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c36 --- Comment #36 from David Richards 2011-08-25 13:25:06 UTC --- @all: Confirmed, complete reformat and reload of OS 11.4 is now scheduled for next Tuesday. We are hopeful that something failed in going from 11.3 -> 11.4. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c38 --- Comment #38 from David Richards 2011-08-25 13:37:49 UTC --- @michal: I'm not sure I would know the best way to do that. Would you maybe dd off the whole disk image to another external disk, and then boot a stand alone distro and then format the drive and then dd it all back? Something like Clonezilla won't work because I believe it would bring back the failure. I have mentioned already too that when we made a physical to virtual copy and put it in VMware, the same types of issues are seen. Vmware downgrades ext4 to ext3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c40 --- Comment #40 from David Richards 2011-08-30 14:41:29 UTC --- @all: We created a Clonezilla of the OS 11.4 image in case we need to debug or test the older version. So today we pulled the disk drives from the RAID array and reseated them, deleted and recreated a new RAID 1+0 array (which has proven to work well on multi-user servers). We then formatted as ext3 and reloaded OS 11.4. It's going to take me all afternoon to restore all of the customizations and test them; hopefully we can move users back on this hardware tomorrow. One issue of note, whatever is wrong with the networking is back already on a fresh install, fully patched from the Internet and then rebooted: linux-gsss:~ # ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72 inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:33726 errors:0 dropped:358 overruns:0 frame:0 Now to re-install everything. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c42 David Richards changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|drichard@largo.com | --- Comment #42 from David Richards 2011-09-01 13:13:50 UTC --- @all: Very sadly I must report that the issue with disk and networking performance has not gone away after the server was completely reformatted and reloaded. I should be able to get 200 users on this hardware. The symptoms are identical. Files that are sitting in /etc have very slow disk access speeds. With 26 users, vi /etc/services blinks for 2 seconds before the file opens. If I copy the file to /tmp or /home, it opens immediately. Tested on other servers with heavier users loads, running SLED 11 and OS 11.3 and this isn't seen at all. I'm sure if we get to our peak loads of 40, that it's going to behave as it did before and be painfully slow. There are two network cards activated just as before, and both are behaving as before. Lots of dropped RX packets. eth0 is the internal network, and eth1 goes out to the Internet. We use this same configuration on all servers with no similar issues on earlier versions of OpenSuse. Does everyone agree that we should move to the 3.0 kernel? I can't even imagine what could be causing these issues. eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72 inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11322212 errors:0 dropped:70978 overruns:0 frame:0 TX packets:11728443 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4811162089 (4588.2 Mb) TX bytes:9917306091 (9457.8 Mb) Interrupt:16 Memory:f8000000-f8012800 eth1 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:74 inet addr:172.23.1.235 Bcast:172.23.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:70739 errors:0 dropped:238 overruns:0 frame:0 TX packets:7314 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15062173 (14.3 Mb) TX bytes:760390 (742.5 Kb) Interrupt:17 Memory:fa000000-fa01280 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c44 --- Comment #44 from Greg Kroah-Hartman 2011-09-01 23:20:23 UTC --- Yes, trying 3.0 would be great, but the number of dropped receive packets is worrisome, and might be indicative of some other type of interrupt routing issue that might be causing the disk delay as well. This hardware did work fine on 11.3, right? So that would kind of rule out the hardware problem, very strange. Yes, trying 3.0 would be a nice test if you can do it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c46 --- Comment #46 from Greg Kroah-Hartman 2011-09-04 16:20:44 UTC --- Wait, the issues followed to a VM clone? The network packet loss and the filesystem delays? On the same exact machine or a different one? I ask about the hardware trying to rule your specific individual machine out from having issues somewhere in it, has this specific machine worked properly with the 11.3 image? Or did you buy it and only try 11.4 on it? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c48 --- Comment #48 from David Richards 2011-09-06 16:49:17 UTC --- @all: I added repository: http://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ and will bring down kernel-desktop 3.0.4-43.1 tonight and get it rebooted tomorrow AM for testing. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c50 --- Comment #50 from David Richards 2011-09-06 17:37:12 UTC --- @Greg: Will do Greg, adjusting everything now. The issue of kernel-desktop vs kernel-default was a bit unknown in my mind because while it is a server, we are using it as a desktop. Will follow your instructions, reboot is scheduled for tomorrow AM. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c52 --- Comment #52 from David Richards 2011-09-07 17:52:36 UTC --- 41 users and still very fast. I'd say for sure the disk issue is resolved in kernel 3.0.4. Dropping lots of packets, here is the current total: ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:1C:C4:93:DF:72 inet addr:128.222.99.243 Bcast:128.222.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:23636204 errors:0 dropped:22611 overruns:0 frame:0 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |drichard@largo.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=712958 https://bugzilla.novell.com/show_bug.cgi?id=712958#c55 Greg Kroah-Hartman changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |RESOLVED InfoProvider|drichard@largo.com | Resolution| |FIXED --- Comment #55 from Greg Kroah-Hartman 2011-09-20 14:50:29 UTC --- Yes, if you want us to work on that issue, it should be a separate bug. Glad the 3.0 kernel is working properly for you, will mark this as fixed now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.