HELP ! YOU-Update on SuSE 8.1 firewall did something eval to my kernel
Hi all, yesterday I updated my SuSE 8.1 system with the recommended (auto) updates through YOU. I noticed that there was a kernel update in the list, but I didn't mind. Today, when under stress, my firewall gives hundres of messages like: Sep 10 11:53:27 proxy1 kernel: NET: 39 messages suppressed. Sep 10 11:53:27 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:27 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:32 proxy1 last message repeated 61 times Sep 10 11:53:32 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:32 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:32 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:38 proxy1 last message repeated 60 times Sep 10 11:53:38 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:38 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:38 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:43 proxy1 last message repeated 59 times Sep 10 11:53:43 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:43 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:43 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:49 proxy1 last message repeated 59 times Sep 10 11:53:49 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:49 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:49 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:52 proxy1 last message repeated 11 times Sep 10 11:53:52 proxy1 kernel: NET: 11 messages suppressed. Sep 10 11:53:52 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:55 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:59 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:59 proxy1 kernel: NET: 1 messages suppressed. Sep 10 11:53:59 proxy1 kernel: Neighbour table overflow. Sep 10 11:54:03 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:54:03 proxy1 kernel: Neighbour table overflow. Sep 10 11:54:03 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:54:08 proxy1 last message repeated 3 times Sep 10 11:54:08 proxy1 kernel: NET: 3 messages suppressed. Sep 10 11:54:08 proxy1 kernel: Neighbour table overflow. I did NOT change a thing besides those updates and rebooted. The firewall is done through iptables and configured with the "shorewall" script which have been in use for over a year now without any problems. Now the firewall simply stops after a certain while. Unfotunately I cannot log in because the SSH process is crashing as well and I am not on site, but I managed to get the logs via email. Any hint / help is appreciated very much. Thank you in advance, Philipp Rusch
On Wed, Sep 10, 2003 at 12:22:18PM +0200, Philipp Rusch wrote:
yesterday I updated my SuSE 8.1 system with the recommended (auto) updates through YOU. I noticed that there was a kernel update in the list, but I didn't mind.
Are you running any applications on that machine? Olaf -- Olaf Kirch | Anyone who has had to work with X.509 has probably okir@suse.de | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann
NO ! okay, yes, squid is running (version 2.4.7) Philipp Olaf Kirch schrieb:
On Wed, Sep 10, 2003 at 12:22:18PM +0200, Philipp Rusch wrote:
yesterday I updated my SuSE 8.1 system with the recommended (auto) updates through YOU. I noticed that there was a kernel update in the list, but I didn't mind.
Are you running any applications on that machine?
Olaf -- Olaf Kirch | Anyone who has had to work with X.509 has probably okir@suse.de | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann
Philipp Rusch wrote:
Hi all,
yesterday I updated my SuSE 8.1 system with the recommended (auto) updates through YOU. I noticed that there was a kernel update in the list, but I didn't mind.
Today, when under stress, my firewall gives hundres of messages like:
Sep 10 11:53:27 proxy1 kernel: NET: 39 messages suppressed.
I did NOT change a thing besides those updates and rebooted. The firewall is done through iptables and configured with the "shorewall" script which have been in use for over a year now without any problems. Now the firewall simply stops after a certain while.
Unfotunately I cannot log in because the SSH process is crashing as well and I am not on site, but I managed to get the logs via email.
Any hint / help is appreciated very much.
This is not a kernel Bug. I would say you've some kind of worm inside your network. Dunno which one exactly but i've seen it on many routers in the last 3 weeks (would say blaster or sobig). The "solution" beside removing the worm is simple: make your arp cache table bigger to hold more arp entrys. This can be done by: echo 1024 > /proc/sys/net/ipv4/neigh/default/gc_thresh1 echo 2048 > /proc/sys/net/ipv4/neigh/default/gc_thresh2 echo 4096 > /proc/sys/net/ipv4/neigh/default/gc_thresh3 this should be ok for now. Hopefully your RAM isn't full at all ;) Your Box will work ok again and the errors should be gone. After that please check for the worm. The worm pings your local network (any ip) and so you'll get many incomplete arp entries. You can check that (if you have access again to the box) with the arp command. If you want, you can track how many entries in your cache by arp -an | wc -l (and you'll see that it increases up to more than 1024, the old default maximum). HTHm SVen
Thanks, great tip ! I will try out the arp cache config immediately. That "worm attack" started on Sept. 5th as I noticed now when investigating further in the logs (working on backwards ...) So nothing that came from the update, since that was done on the 9th. Hmmm, double checked the complete LAN (ca. 58 NT4 boxes) with McAfee 4291 DAT (most recent) and found nothing suspicious yesterday. Although we are not done with the RPC patches at all workstations. Is this an udp port that ms-blaster is using ? and I thought it uses only tcp port 135 or 445 (smb-shares ?) All I open up in my firewall config is port 22 (ssh) and the ports 8080/8090 and 3128 for the different proxy setups (historical). Then there are some 515 (LAN-printers from company Intranet), but that's all ... there should be some DROPS or REJECTS from the firewall, but I didn't notice such ... Still searching, Philipp Sven 'Darkman' Michels schrieb:
Philipp Rusch wrote:
Hi all,
yesterday I updated my SuSE 8.1 system with the recommended (auto) updates through YOU. I noticed that there was a kernel update in the list, but I didn't mind.
Today, when under stress, my firewall gives hundres of messages like:
Sep 10 11:53:27 proxy1 kernel: NET: 39 messages suppressed.
I did NOT change a thing besides those updates and rebooted. The firewall is done through iptables and configured with the "shorewall" script which have been in use for over a year now without any problems. Now the firewall simply stops after a certain while.
Unfotunately I cannot log in because the SSH process is crashing as well and I am not on site, but I managed to get the logs via email.
Any hint / help is appreciated very much.
This is not a kernel Bug. I would say you've some kind of worm inside your network. Dunno which one exactly but i've seen it on many routers in the last 3 weeks (would say blaster or sobig). The "solution" beside removing the worm is simple: make your arp cache table bigger to hold more arp entrys. This can be done by: echo 1024 > /proc/sys/net/ipv4/neigh/default/gc_thresh1 echo 2048 > /proc/sys/net/ipv4/neigh/default/gc_thresh2 echo 4096 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
this should be ok for now. Hopefully your RAM isn't full at all ;) Your Box will work ok again and the errors should be gone. After that please check for the worm. The worm pings your local network (any ip) and so you'll get many incomplete arp entries. You can check that (if you have access again to the box) with the arp command. If you want, you can track how many entries in your cache by arp -an | wc -l (and you'll see that it increases up to more than 1024, the old default maximum).
HTHm SVen
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
I solved it ! Problem was a hardware defect of the network card connecting to the "outside" of the firewall machine, somehow the mac address was incomplete in packets coming from this interface. We have an identical standby machine at that site that we reconfigured to act as the main proxy/firewall and everything works fine now. Thanks for all replies, Philipp Philipp Rusch schrieb:
Hi all,
yesterday I updated my SuSE 8.1 system with the recommended (auto) updates through YOU. I noticed that there was a kernel update in the list, but I didn't mind.
Today, when under stress, my firewall gives hundres of messages like:
Sep 10 11:53:27 proxy1 kernel: NET: 39 messages suppressed. Sep 10 11:53:27 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:27 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:32 proxy1 last message repeated 61 times Sep 10 11:53:32 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:32 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:32 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:38 proxy1 last message repeated 60 times Sep 10 11:53:38 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:38 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:38 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:43 proxy1 last message repeated 59 times Sep 10 11:53:43 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:43 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:43 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:49 proxy1 last message repeated 59 times Sep 10 11:53:49 proxy1 kernel: NET: 59 messages suppressed. Sep 10 11:53:49 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:49 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:52 proxy1 last message repeated 11 times Sep 10 11:53:52 proxy1 kernel: NET: 11 messages suppressed. Sep 10 11:53:52 proxy1 kernel: Neighbour table overflow. Sep 10 11:53:55 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:59 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:53:59 proxy1 kernel: NET: 1 messages suppressed. Sep 10 11:53:59 proxy1 kernel: Neighbour table overflow. Sep 10 11:54:03 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:54:03 proxy1 kernel: Neighbour table overflow. Sep 10 11:54:03 proxy1 kernel: neigh_alloc(): neighbour table flood for neigh_table c0329840 Sep 10 11:54:08 proxy1 last message repeated 3 times Sep 10 11:54:08 proxy1 kernel: NET: 3 messages suppressed. Sep 10 11:54:08 proxy1 kernel: Neighbour table overflow.
I did NOT change a thing besides those updates and rebooted. The firewall is done through iptables and configured with the "shorewall" script which have been in use for over a year now without any problems. Now the firewall simply stops after a certain while.
Unfotunately I cannot log in because the SSH process is crashing as well and I am not on site, but I managed to get the logs via email.
Any hint / help is appreciated very much.
Thank you in advance, Philipp Rusch
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
participants (3)
-
Olaf Kirch -
Philipp Rusch -
Sven 'Darkman' Michels