Re: [suse-security] Apache upgrade from SuSE ?
Hi Oliver,
the problem with the apache upgrade seems to be that the stock apache from SuSE 7.0 was 1.3.12 (IIRC, it was definately <1.3.19).
Yes, that's right.
Then, on 28.April 2001, SuSe seems to have issued a apache 1.3.19 for suse linux 7.0, at least that is what I figure out from the fact that
Yes.
http://www.suse.de/de/support/download/updates/70_i386.html
contains links to mod_contrib.rpm, mod_perl.rpm and other modules for apache 1.3.19.
Strictly speaking those module updates should not have been necessary.
I doubt that, from my own experiences.
The module API has not changed since back 1.3.12 days. However, there are some other package interdependencies as well -- (if you ever typed ldd /usr/lib/apache/libphp4.so you know what I mean...) so if for example openldap has been updated meanwhile as well and the API used by php has actually changed, you might end up updating a rebuilt php package as well. I.e. a package like mod_php might have to rebuilt quite often. Now the problem is not, for us, to track down these build dependencies, but to cut them down by translating them into a set of packages that need rebuilding/updating (on your system!) or not. It's very complex, unfortunately.
I know of the problems, and I think your strategy, as also used with the openssh issue, is the right one.
Irritatingly, there's no link to an apache 1.3.19 upgrade from that time, maybe you people deleted that when the new apache came out.
Do you mean the symbolic link named apache.rpm? It always points to the current rpm for convenience, so it has been changed since then.
No, I mean a link and a note at http://www.suse.de/de/support/download/updates/70_i386.html As you see, there are said modules listed at 28.April 2001, but not apache itself. Before I upgraded, I grepped this page for other occurrencies of the word "apache" and found nothing else. So I concluded that apache hasn't been updated before.
Now came the new upgrade, and only mod_ssl and apache itself were published. People like me upgraded, got no rpm dependency problems (yes, without using --nodeps), but apache segfaulted on every request.
As the same procedure has worked for many people, I assume that there is something in your setup. A special set of modules or something else.
Maybe the other people had done the previous update, I didn't, and at least two people on the mailing list didn't also and had seemingly the same problem.
Important: we assume that people have installed the previous security updates of apache. I hope that this is a fair assumption?
Not quite, as you see now. I don't know what the issue was on April, 28th 2001, and I don't find a security anouncement from SuSE concerning apache from that time. But I'm quite sure that _if_ it had been an issue for my servers, I had upgraded apache - but I hadn't. Look at the last openssh issue, I think it's reasonable _not_ to assume that everyone will update openssh - it's just not an issue for people with most configurations. If, in the future, SuSE updates any package which needs the new openssh (not very likely, I know) on 7.0, the same thing could happen again.
It took me some time to figure out that my apache modules were the culprits, because I _had_ checked for a previous apache upgrade which I might have omitted for some reason - and, as I said, rpm didn't complain.
You had the modules from stock 7.0 installed (hope I got you right)? They should actually work with the current apache, with the exception of mod_ssl. Do you know which specific module failed?
After I updated mod_contrib and mod_perl, the servers ran alright - before that, segfaults on every request. But the server started up ok. Another data point: even using the src.rpm didn't help. Ok, one thing, I had mod_gzip compiled for apache 1.3.12. But I had tried to get a working apache without that - it didn't work. To elaborate, I commented out everything in httpd.conf which is related to mod_gzip. It didn't work, but after updating the two modules, itd did, in the exact same configuration.
Sometimes, when we can't avoid to make version updates of packages (and apache 1.3.12->1.3.19 is such a candidate) it can't be guaranteed that the build procedure can actually be the same.
Two examples:
1) Newer apache versions (>1.3.22 or so) use an external (system installed) expat library in favor of the internal one, to avoid library symbol name clashes when a module (say, mod_perl) loads the external expat lib into the same namespace.
2) With apache 1.3.23 or so and later the accept serialization method has been made runtime configurable. The configuration style has changed.
Simply and boldly offering apache 1.3.26 updates for all distributions would be prone to break a lot of things because of changes like that. It would be hard to get it right again, so everything works still as expected, the configuration files are still valid, and so on.
It's absolutely no question for me that staying with the same version number and backporting security patches is the right thing to do under almost any circumstances. Esp. with apache, where there may be custom modules which depend on that very version.
It would have been helpful if the advisory contained a hint about updating the apache modules, because the described update strategy _must_ fail on a stock 7.0.
As I understand it, we would need a special hint for users that have never installed one of the previous apache updates before.
Wrt the texts of the module update packages, they say "update package for apache 1.3.19". Which is what they ought to say.
This is correct, but if it's true that one needs to have other, older updates to get a working server, it would be a great help to describe that in the advisory. What happend was: stock 7.0 -> apache+mod_ssl update from 18.Jun 2002 -> crash what I had to do was: stock 7.0 -> mod_contrib+mod_perl from 28.April 2001 + apache+mod_ssl from 18.Jun -> works IOW, the update from 18.Jun was dependend on the update from 28.April 2001, and that was neither mentioned in the advisory nor reflected in the rpm-dependencies.
Mind you, I'm not complaining, just trying to give a hint where these problems might come from.
We are greatful for such help!
And you are doing a great job, esp. with the last two big "incidents". cheers, oliver
Hi, On Wed, Jul 03, 2002 at 12:02:47PM +0200, Oliver Bleutgen wrote:
Strictly speaking those module updates should not have been necessary.
I doubt that, from my own experiences.
Well I can not state that it works for all ~60 modules -- but for mod_php4 and mod_perl it does.
No, I mean a link and a note at http://www.suse.de/de/support/download/updates/70_i386.html
As you see, there are said modules listed at 28.April 2001, but not apache itself. Before I upgraded, I grepped this page for other occurrencies of the word "apache" and found nothing else. So I concluded that apache hasn't been updated before.
You are right -- the older entries are superseded (i.e. removed) on that web site (which is bad IMO). Also, looking on http://www.suse.de/de/support/security/ there is only one apache announcement-- the last one. Apparently, we did not issue an announcement at that time -- which would be bad IMO but I guess we might not have had the same infrastructure at that time compared to now). Or (!) we did issue the information about apache, but not in its own announcement but in a 2) pending vulnerabilities, solutions, workarounds section. This way, the information is hard to find because the website does list only the titles, not the affected package names, nor offers it a way to search through the announcements. This is bad IMO as well. I'll stimulate discussion about what kind of improvements would be useful here. Be assured that your criticism is very helpful.
Now came the new upgrade, and only mod_ssl and apache itself were published. People like me upgraded, got no rpm dependency problems (yes, without using --nodeps), but apache segfaulted on every request.
I overlooked that you did not need --nodeps in your last mail. That's interesting. Hhm, most module packages of 7.0 did just have a Requires on apache, so that's fine (even better).
As the same procedure has worked for many people, I assume that there is something in your setup. A special set of modules or something else.
Maybe the other people had done the previous update, I didn't, and at least two people on the mailing list didn't also and had seemingly the same problem.
Important: we assume that people have installed the previous security updates of apache. I hope that this is a fair assumption?
Not quite, as you see now. I don't know what the issue was on April, 28th 2001, and I don't find a security anouncement from SuSE concerning apache from that time. But I'm quite sure that _if_ it had been an issue for my servers, I had upgraded apache - but I hadn't. Look at the last openssh issue, I think it's reasonable _not_ to assume that everyone will update openssh - it's just not an issue for people with most configurations.
Well, openssh was a special case -- there haven't been any such dodgy apache updates.
If, in the future, SuSE updates any package which needs the new openssh (not very likely, I know) on 7.0, the same thing could happen again.
It's hard to take everything into account...
It took me some time to figure out that my apache modules were the culprits, because I _had_ checked for a previous apache upgrade which I might have omitted for some reason - and, as I said, rpm didn't complain.
You had the modules from stock 7.0 installed (hope I got you right)? They should actually work with the current apache, with the exception of mod_ssl. Do you know which specific module failed?
After I updated mod_contrib and mod_perl, the servers ran alright - before that, segfaults on every request. But the server started up ok. Another data point: even using the src.rpm didn't help. Ok, one thing, I had mod_gzip compiled for apache 1.3.12. But I had tried to get a working apache without that - it didn't work. To elaborate, I commented out everything in httpd.conf which is related to mod_gzip. It didn't work, but after updating the two modules, itd did, in the exact same configuration.
It's hard to tell anything without getting hands on the system -- without knowing the configuration and so on...
It's absolutely no question for me that staying with the same version number and backporting security patches is the right thing to do under almost any circumstances. Esp. with apache, where there may be custom modules which depend on that very version.
It would have been helpful if the advisory contained a hint about updating the apache modules, because the described update strategy _must_ fail on a stock 7.0.
As I understand it, we would need a special hint for users that have never installed one of the previous apache updates before.
Wrt the texts of the module update packages, they say "update package for apache 1.3.19". Which is what they ought to say.
This is correct, but if it's true that one needs to have other, older updates to get a working server, it would be a great help to describe that in the advisory.
What happend was: stock 7.0 -> apache+mod_ssl update from 18.Jun 2002 -> crash
what I had to do was: stock 7.0 -> mod_contrib+mod_perl from 28.April 2001 + apache+mod_ssl from 18.Jun -> works
IOW, the update from 18.Jun was dependend on the update from 28.April 2001, and that was neither mentioned in the advisory nor reflected in the rpm-dependencies.
It looks so. I don't believe it, but it's well possible. We might dig into it further, but I can't promise anything. Thanks for the details, since now we know what to look for. Peter -- VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
On Wed, 03 Jul 2002, Oliver Bleutgen wrote:
Then, on 28.April 2001, SuSe seems to have issued a apache 1.3.19 for suse linux 7.0, at least that is what I figure out from the fact that
Yes.
Maybe the other people had done the previous update, I didn't, and at least two people on the mailing list didn't also and had seemingly the same problem.
Important: we assume that people have installed the previous security updates of apache. I hope that this is a fair assumption?
Not quite, as you see now. I don't know what the issue was on April, 28th 2001, and I don't find a security anouncement from SuSE concerning apache from that time. But I'm quite sure that _if_ it had been an issue for my servers, I had upgraded apache - but I hadn't.
... quotes cruelly edited by dproc ... No-one should forget the other reason why earlier updates are not installed ... a fresh install from the original media. Then it would be normal, I think, for me to go to the critical updates webpage (or possibly run YOU against the updates directory) and install the latest update packages only. I think I would have missed the need for the sequential updates, as Oliver did , if I had done this. Am I right? (Still it took another much bigger software company many years of customer complaints to produce a tool to help install their 'hotfixes' in the correct order, and their product was much smaller that 7 CDs. So, like Oliver, I am not complaining.)
Wrt the texts of the module update packages, they say "update package for apache 1.3.19". Which is what they ought to say.
This is correct, but if it's true that one needs to have other, older updates to get a working server, it would be a great help to describe that in the advisory.
What happend was: stock 7.0 -> apache+mod_ssl update from 18.Jun 2002 -> crash
what I had to do was: stock 7.0 -> mod_contrib+mod_perl from 28.April 2001 + apache+mod_ssl from 18.Jun -> works
IOW, the update from 18.Jun was dependend on the update from 28.April 2001, and that was neither mentioned in the advisory nor reflected in the rpm-dependencies.
On Wed, Jul 03, 2002 at 01:52:56PM -0500, dproc@dol.net wrote:
No-one should forget the other reason why earlier updates are not installed ... a fresh install from the original media. Then it would be normal, I think, for me to go to the critical updates webpage (or possibly run YOU against the updates directory) and install the latest update packages only.
I think I would have missed the need for the sequential updates, as Oliver did , if I had done this. Am I right?
Totally and utterly right, yes. *Sigh* :) Well, I doubt that the necessity of sequential updates has been the problem here -- after all, that one can be tested most easily. (Just make plain install from CDs, ...) But ensuring that updating from any possible other level works might be more of a challenge... But still, a package that has been rebuilt or a package with some nonintrusive patch added usually is no problem at all. But there could be packaging errors, version updates might be nesessary for various reasons, ... The worrying thing is that the simple addition of a real small patch to a package like this can lead to a brutal confrontation with the past ;) Peter -- VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
On Wed, Jul 03, 2002 at 12:02:47PM +0200, Oliver Bleutgen wrote:
Simply and boldly offering apache 1.3.26 updates for all distributions would be prone to break a lot of things because of changes like that. It would be hard to get it right again, so everything works still as expected, the configuration files are still valid, and so on.
It's absolutely no question for me that staying with the same version number and backporting security patches is the right thing to do under almost any circumstances. Esp. with apache, where there may be custom modules which depend on that very version.
Just an example --because we talked about exactly this-- which I just read on the httpd list:
I am writing in hopes that you can help us with an urgent problem we are having with a bug fix you put into Apache 1.3.26 I have spent two days tracking this down and am certain the issue is with your fix.
Due to an error in OUR online game code, we were incorrectly requesting files using 'HTTP-1.0' instead of 'HTTP/1.0' on the GET request line. As you know, this is wrong. However, suprisingly, this worked just fine for several years with both Apache and other Web servers, presumably because the server just ignored it or defaulted to HTTP/1.0. If you want to test, try our down-level Apache server at lobby.dqsoft.com with GET /index.html HTTP-1.0 I am sure I am not the only one with this problem, as there are several socket tutorials and such that incorrectly say 'HTTP-1.0'.
q.e.d. Peter -- VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
participants (3)
-
dproc＠dol.net -
Oliver Bleutgen -
Peter Poeml