So, is Suse working on a fix for the mysqld problem?
Unfortunately, I didn't read that thread until now and updated all of our machines to kernel 2.4.21-238 yesterday and hit the mysql problem already discussed on the weekend. Since then the thread has been quiet. So, is Suse working on a fix for this? After all, mysqld processes maxing out is a *serious* problem, the systems are effectively DoSing themselves. I urge Suse to issue a new announcement that warns about this problem, so vendors are warned and can consider halting on the upgrade. What I know so far is: - it seems all 2.4.21-238 (no matter, if Athlon, Intel or smp) on all newer Suse systems (we have 8.1 and 9.0 at the moment) are affected (maybe only 32bit, we don't have any 64bit). All systems run with latest standard Suse programs. - each time a mysql connection is closed that mysqld process goes <defunct> and stays forever until killed with "killall -9 mysqld", it's sufficient to "mysql -pxxx" and just quit, so it is surely not a problem of corrupted databases or specific programs - mysql-max does *not* seem to be affected - I have one system (Suse 9.0) with normal mysqld which doesn't seem to be affected. I don't know why. But all other 9.0 systems are affected. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
On Wed, Aug 11, 2004 at 01:24:12PM +0200, Kai Schaetzl wrote:
Unfortunately, I didn't read that thread until now and updated all of our machines to kernel 2.4.21-238 yesterday and hit the mysql problem already discussed on the weekend. Since then the thread has been quiet. So, is Suse working on a fix for this? After all, mysqld processes maxing out is a *serious* problem, the systems are effectively DoSing themselves. I urge Suse to issue a new announcement that warns about this problem, so vendors are warned and can consider halting on the upgrade.
What I know so far is: - it seems all 2.4.21-238 (no matter, if Athlon, Intel or smp) on all newer Suse systems (we have 8.1 and 9.0 at the moment) are affected (maybe only 32bit, we don't have any 64bit). All systems run with latest standard Suse programs. - each time a mysql connection is closed that mysqld process goes <defunct> and stays forever until killed with "killall -9 mysqld", it's sufficient to "mysql -pxxx" and just quit, so it is surely not a problem of corrupted databases or specific programs - mysql-max does *not* seem to be affected - I have one system (Suse 9.0) with normal mysqld which doesn't seem to be affected. I don't know why. But all other 9.0 systems are affected.
We are working on it. Ciao, Marcus
Marcus Meissner wrote on Wed, 11 Aug 2004 13:32:28 +0200:
- mysql-max does *not* seem to be affected
seems to be wrong. Was a first observation on all the systems which weren't affected. I changed to mysqld-max on an affected system and it's still "defuncting".
We are working on it.
Good to know. Any quick workaround until you roll it out? Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
On Wed, Aug 11, 2004 at 03:02:46PM +0200, Kai Schaetzl wrote:
Marcus Meissner wrote on Wed, 11 Aug 2004 13:32:28 +0200:
- mysql-max does *not* seem to be affected
seems to be wrong. Was a first observation on all the systems which weren't affected. I changed to mysqld-max on an affected system and it's still "defuncting".
We are working on it.
Good to know. Any quick workaround until you roll it out?
No workaround sorry. We are trying to get fixed kernels out today. Ciao, Marcus
After stopping the mysql deamon I have reapplied the kernel patch on several affected machines. The zombie problem doesn't exist any longer. Karsten
-----Original Message----- From: Kai Schaetzl [mailto:maillists@conactive.com] Sent: Wednesday, August 11, 2004 1:24 PM To: suse-security@suse.com Subject: [suse-security] So, is Suse working on a fix for the mysqld problem?
Unfortunately, I didn't read that thread until now and updated all of our machines to kernel 2.4.21-238 yesterday and hit the mysql problem already discussed on the weekend. Since then the thread has been quiet. So, is Suse working on a fix for this? After all, mysqld processes maxing out is a *serious* problem, the systems are effectively DoSing themselves. I urge Suse to issue a new announcement that warns about this problem, so vendors are warned and can consider halting on the upgrade.
What I know so far is: - it seems all 2.4.21-238 (no matter, if Athlon, Intel or smp) on all newer Suse systems (we have 8.1 and 9.0 at the moment) are affected (maybe only 32bit, we don't have any 64bit). All systems run with latest standard Suse programs. - each time a mysql connection is closed that mysqld process goes <defunct> and stays forever until killed with "killall -9 mysqld", it's sufficient to "mysql -pxxx" and just quit, so it is surely not a problem of corrupted databases or specific programs - mysql-max does *not* seem to be affected - I have one system (Suse 9.0) with normal mysqld which doesn't seem to be affected. I don't know why. But all other 9.0 systems are affected.
Kai
--
Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
Karsten Lehmann wrote on Wed, 11 Aug 2004 13:55:47 +0200:
After stopping the mysql deamon I have reapplied the kernel patch on several affected machines. The zombie problem doesn't exist any longer.
which OS? I tried it on one 9.0 machine and there's no difference. That machine had actually no mysqld running at all (and never before) during the first installation. I started it and it was even worse on that system because it produced a series of defunct processes right at startup, so that the startup takes so long that the init script thinks it failed. I reapplied the kernel while mysqld was shutdown, rebooted and there's unfortunately no difference :-( Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
After stopping the mysql deamon I have reapplied the kernel patch on several affected machines. The zombie problem doesn't exist any longer.
That's a (happy) coincidence, the error/malfunction appears randomly due to an uninitialized variable. The problem is fixed, and the kernels are on their way to your YOU. Please be patient while we're makeing sure that you wouldn't even be able to write mails about failures any more...
which OS? I tried it on one 9.0 machine and there's no difference. That machine had actually no mysqld running at all (and never before) during the first installation. I started it and it was even worse on that system because it produced a series of defunct processes right at startup, so that the startup takes so long that the init script thinks it failed. I reapplied the kernel while mysqld was shutdown, rebooted and there's unfortunately no difference :-(
Thanks,
Roman.
--
- -
| Roman Drahtmüller
THAT would be nice....
----- Original Message -----
From: "Roman Drahtmueller"
After stopping the mysql deamon I have reapplied the kernel patch on
several
affected machines. The zombie problem doesn't exist any longer.
That's a (happy) coincidence, the error/malfunction appears randomly due to an uninitialized variable. The problem is fixed, and the kernels are on their way to your YOU. Please be patient while we're makeing sure that you wouldn't even be able to write mails about failures any more...
Roman Drahtmueller wrote on Wed, 11 Aug 2004 15:30:39 +0200 (MEST):
The problem is fixed, and the kernels are on their way to your YOU.
I installed the 241 kernel on 8.1 and it fixed the problem. Many thanks! I'm now waiting for the 9.0 kernel so I can test it as well. It's not yet on ftp.suse.com. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
Why does the new 9.0 kernel rpm have the same name as the old one? That effectively disables automatic online updates, f.i. fou4s won't work with this update. I don't think that's desirable, many installations will not be able to identify this update. It's also not in line with the new 8.0 kernel rpms which changed names. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
Kai Schaetzl wrote on Wed, 11 Aug 2004 22:43:16 +0200:
Why does the new 9.0 kernel rpm have the same name as the old one? That effectively disables automatic online updates, f.i. fou4s won't work with this update. I don't think that's desirable, many installations will not be able to identify this update. It's also not in line with the new 8.0 kernel rpms which changed names.
It seems the 9.0 update is completely hosed. That *is* the old file despite the newer filedate and what the info file says. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
Why does the new 9.0 kernel rpm have the same name as the old one? That effectively disables automatic online updates, f.i. fou4s won't work with this update. I don't think that's desirable, many installations will not be able to identify this update. It's also not in line with the new 8.0 kernel rpms which changed names.
8.0? Read 9.1? (?)
It seems the 9.0 update is completely hosed. That *is* the old file despite the newer filedate and what the info file says.
An intermediate kernel - it shouldn't be cought by YOU/fou4s, _because_ it is not the complete fix yet. 8.1 and 8.2 are out already, 9.0 is being worked on, still. The 2.6 kernel on 9.1 doesn't suffer from the stalls of mysql and other threaded applications. Thanks, Roman.
Roman Drahtmueller wrote on Thu, 12 Aug 2004 04:02:46 +0200 (MEST): Hi Romain,
8.0? Read 9.1?
8.*, 9.0: old kernel: 2.4.21-238 8.*: new kernel: 2.4.21-241 9.0: (seemingly) new kernel: 2.4.21-238 I meant to say that this is confusing, at best.
An intermediate kernel - it shouldn't be cought by YOU/fou4s, _because_ it is not the complete fix yet. 8.1 and 8.2 are out already, 9.0 is being worked on, still.
My observations: ftp://ftp.suse.com/pub/suse/i386/update/9.0/rpm/i586/k_deflt-2.4.21-238.i586.rpm Aug 11 09:32 this pretends to be a new file by the date shown, it isn't ftp://ftp.suse.com/pub/suse/i386/update/9.0/rpm/i586/k_athlon-2.4.21-238.i586_en info Aug 11 16:50 Description: This update fixes a problem of the earlier fix for the signal queuing DoS with threaded applications, which could occasionaly leave zombie threads. new patch info file: ftp://ftp.suse.com/pub/suse/i386/update/9.0/patches/kernel-51570/ This update fixes a problem of the earlier fix for the signal queuing DoS with threaded applications, which could occasionaly leave zombie threads. None of this is true and the rpm file dated Aug 11 09:32 per the web page is actually identical with the rpm I downloaded on Aug 10. Would you agree that something's wrong here? If you are still working on the fix then you shouldn't push out new documentation which says otherwise and is incorrect. If that is an intermediate kernel which should partly fix the problem I wonder why it is identical with the kernel dated July 30 I got via fou4s on Aug 10 (md5sum is 2f0b04a5a541dc7e968e34cb673212a0). Nothing of this fits together. It looks to me like you pushed out some information too early or someone hosed the replacement of the rpm and put the old rpm with a new date in place. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de & http://msie.winware.org
On Thu, Aug 12, 2004 at 01:51:36PM +0200, Kai Schaetzl wrote:
Roman Drahtmueller wrote on Thu, 12 Aug 2004 04:02:46 +0200 (MEST):
Hi Romain,
8.0? Read 9.1?
8.*, 9.0: old kernel: 2.4.21-238 8.*: new kernel: 2.4.21-241 9.0: (seemingly) new kernel: 2.4.21-238
I meant to say that this is confusing, at best.
An intermediate kernel - it shouldn't be cought by YOU/fou4s, _because_ it is not the complete fix yet. 8.1 and 8.2 are out already, 9.0 is being worked on, still.
My observations: ftp://ftp.suse.com/pub/suse/i386/update/9.0/rpm/i586/k_deflt-2.4.21-238.i586.rpm Aug 11 09:32 this pretends to be a new file by the date shown, it isn't
ftp://ftp.suse.com/pub/suse/i386/update/9.0/rpm/i586/k_athlon-2.4.21-238.i586_en info Aug 11 16:50 Description: This update fixes a problem of the earlier fix for the signal queuing DoS with threaded applications, which could occasionaly leave zombie threads.
new patch info file: ftp://ftp.suse.com/pub/suse/i386/update/9.0/patches/kernel-51570/ This update fixes a problem of the earlier fix for the signal queuing DoS with threaded applications, which could occasionaly leave zombie threads.
None of this is true and the rpm file dated Aug 11 09:32 per the web page is actually identical with the rpm I downloaded on Aug 10.
Would you agree that something's wrong here? If you are still working on the fix then you shouldn't push out new documentation which says otherwise and is incorrect. If that is an intermediate kernel which should partly fix the problem I wonder why it is identical with the kernel dated July 30 I got via fou4s on Aug 10 (md5sum is 2f0b04a5a541dc7e968e34cb673212a0).
Nothing of this fits together. It looks to me like you pushed out some information too early or someone hosed the replacement of the rpm and put the old rpm with a new date in place.
The 9.0 kernel we released yesterday does not contain the fix due to a small technical problem here. All others kernels did contain the bugfix. We will try to push out the fixed 9.0 kernel as soon as possible. Ciao, Marcus
Roman Drahtmueller wrote:
8.1 and 8.2 are out already, 9.0 is being worked on, still. The 2.6 kernel on 9.1 doesn't suffer from the stalls of mysql and other threaded applications.
From what I understand, the latest 8.0 kernels (k_*-2.4.18-310) that you guys kindly published are affected by the problem as well.
If so, will you bring out a new 8.0 kernel as well, even if 8.0 has reached its EOL? Thanks -- Till -- Dipl.-Inform. Till Dörges PRESECURE (R) Researcher Consulting GmbH Phone: +49 (0)700 / PRESECURE td@pre-secure.de A daily view on Internet Attacks https://www.ecsirt.net/sensornet
8.1 and 8.2 are out already, 9.0 is being worked on, still. The 2.6 kernel on 9.1 doesn't suffer from the stalls of mysql and other threaded applications.
From what I understand, the latest 8.0 kernels (k_*-2.4.18-310) that you guys kindly published are affected by the problem as well.
If so, will you bring out a new 8.0 kernel as well, even if 8.0 has reached its EOL?
The latter. There are some few packages in the queue for 8.0, but they are just waiting for their publication date and have been ready for quite some time... It should be possible, however, to run an 8.1 kernel on an 8.0 system - and the 8.1 kernel is an extremely reliable one. I haven't seen a crash so far. Roman.
participants (6)
-
Julio C. Esquivel -
Kai Schaetzl -
Karsten Lehmann -
Marcus Meissner -
Roman Drahtmueller -
Till Dörges