[Bug 214863] New: missing <IfModule> in /etc/apache2/server-tuning.conf and a strange apache exit
https://bugzilla.novell.com/show_bug.cgi?id=214863 Summary: missing <IfModule> in /etc/apache2/server-tuning.conf and a strange apache exit Product: SUSE Linux 10.1 Version: Final Platform: Other OS/Version: Other Status: NEW Severity: Critical Priority: P5 - None Component: Network AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: suse-beta@cboltz.de QAContact: qa@suse.de This report covers two related issues - the first is quite obvious, the second isn't :-/ and caused apache to go down hard :-( First: error_log: Syntax error on line 101 of /etc/apache2/server-tuning.conf: Invalid command 'BrowserMatch', perhaps misspelled or defined by a module not included in the server configuration The BrowserMatch directive comes from mod_setenvif and should be encapsulated in a <IfModule> section. Second: /etc/sysconfig/apache2 contains APACHE_MODULES="apparmor actions alias auth_basic authn_file authz_host authz_groupfile authz_default authz_user authn_dbm autoindex cgi dir env expires include log_config mime negotiation setenvif ssl suexec userdir php5 rewrite proxy proxy_http" Please notice that "setenvif" is included in this list, but its BrowserMatch directive was reported as unknown. Unfortunately, this bug caused apache to go down while it was reload'ed by logrotate - this is the reason why I mark this bug critical. To make it even more strange: logrotate called "rcapache reload" several times. Most times it worked, but one time it killed apache :-( Needless to say that I did not change any config file while logrotate run. I'll attach a (commented) error_log sniplet. As far as I can tell, the relevant error message is [error] (9)Bad file descriptor: apr_socket_accept: (client socket) It appeared 5 times on the failed reload. However, it also appears once on every successful reload. (Google found only some people wondering about this message, but no explanation what's wrong.) BTW: The apache error_log seems to be the only one reporting this problem. There's nothing in /var/log/messages about this (like OOM killer) and my status monitoring also doesn't show strange values like high load or high traffic - the only exception is of course the number of running apache processes. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #1 from suse-beta@cboltz.de 2006-10-24 18:51 MST ------- Created an attachment (id=102513) --> (https://bugzilla.novell.com/attachment.cgi?id=102513&action=view) /var/log/apache2/error_log sniplet, with some comments added -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 mhorvath@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team- |poeml@novell.com |screening@forge.provo.novell| |.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 mhorvath@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |suse-beta@cboltz.de ------- Comment #2 from poeml@novell.com 2006-10-27 10:14 MST ------- Please attach /etc/sysconfig/apache2, /etc/apache2/sysconfig.d/* and /etc/apache2/httpd.conf. Or send me your complete configuration, minus ssl certificats, in private. The messages are harmless. And that logrotate reloads apache several times is normal (once per logfile). And the syntax error is there because... there's a syntax erorr :) Let's look at your config. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 suse-beta@cboltz.de changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|suse-beta@cboltz.de | ------- Comment #3 from suse-beta@cboltz.de 2006-10-27 13:39 MST ------- (In reply to comment #2)
Or send me your complete configuration, minus ssl certificats, in private.
I'll do so - you'll receive a tarball in some minutes
The messages are harmless.
BTW: What do they mean? Google only found questions, no answers ;-)
And that logrotate reloads apache several times is normal (once per logfile).
That is/was my understanding also.
And the syntax error is there because... there's a syntax erorr :) Let's look at your config.
I still wonder why the other reloads of the very same config did not show up a warning about syntax errors... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 suse-beta@cboltz.de changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Critical |Blocker ------- Comment #4 from suse-beta@cboltz.de 2006-11-04 16:20 MST ------- Apache exited again :-( last lines from error_log: [Fri Nov 03 23:38:11 2006] [error] [client 212.241.203.109] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:) [Sat Nov 04 23:17:03 2006] [notice] Graceful restart requested, doing restart [Sat Nov 04 23:17:03 2006] [error] (9)Bad file descriptor: apr_socket_accept: (client socket) Syntax error on line 4 of /etc/apache2/conf.d/phpmyadmin.conf: Invalid command 'php_admin_value', perhaps misspelled or defined by a module not included in the server configuration Please notice that a) the "Graceful restart requested" was the first error_log entry today (well, the vHosts have separate error_logs, but those contain only some "file not found" messages) b) this time, it complained about php_admin_value - it seems the not available module changes randomly... c) the crash happened on the first reload this time BTW: Did you find something suspicious in my config files? (raising severity because this is more or less reproducable - and setting up a "rcapache2 status || rcapache2 restart" cronjob to avoid more downtime :-/ ) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |suse-beta@cboltz.de ------- Comment #5 from poeml@novell.com 2006-11-06 06:05 MST ------- I didn't receive mail with your configuration. When did you send it? BTW, if it happens again, please grab the output of 'a2enmod -l' before and after 'rcapache2 restart'. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Blocker |Critical ------- Comment #6 from poeml@novell.com 2006-11-06 06:11 MST ------- Btw, the fact that this is reproducable on your machine doesn't make this a blocker. :-) Not diminuishing the problem it causes you, and I'll do anything to quickly help you, but the apache from 10.1 runs fine for thousands of people, so there clearly is something wrong in your particular setup. For example, if you are scripting vhost configuration or other stuff, or play with apparmor, there are a lot of things that could couse mistakes. For instance, one thing that smells fishy is that you say "apache dies", although on reload it _never_ dies due to invalid configuration, because rcapache2 _always_ checks the config before attempting a reload and refuses to send _any_ signal to the running apache if the configuration doesn't work. Please check your setup again. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #7 from poeml@novell.com 2006-11-06 06:13 MST ------- BTW, did you look into /var/log/audit/audit.log? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 suse-beta@cboltz.de changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|suse-beta@cboltz.de | ------- Comment #8 from suse-beta@cboltz.de 2006-11-06 15:03 MST ------- (In reply to comment #5)
I didn't receive mail with your configuration. When did you send it?
From: Christian Boltz
BTW, if it happens again, please grab the output of 'a2enmod -l' before and after 'rcapache2 restart'.
added to my keepalive cronjob. (In reply to comment #6)
Btw, the fact that this is reproducable on your machine doesn't make this a blocker. :-)
OK, accepted ;-)
For example, if you are scripting vhost configuration or other stuff, or
I indeed do this - but have checked at least some of the generated files and think my script works well.
play with apparmor,
Well, I wouldn't call it a game *g*
For instance, one thing that smells fishy is that you say "apache dies", although on reload it _never_ dies due to invalid configuration, because rcapache2 _always_ checks the config before attempting a reload and refuses to send _any_ signal to the running apache if the configuration doesn't work.
That's how I understand the apache initscript also - but something[tm] seems to be broken somewhere[tm]. I'm quite sure it is _not_ broken config because a simple restart (without config changes) makes it working again. I would more think of some type of race condition - it reads the config before all modules are loaded completely or something like that. (In reply to comment #7)
BTW, did you look into /var/log/audit/audit.log?
yes, regularly - there's nothing suspicious in it. Especially, I didn't notice anything that could be related to the apache crash. I also regularly update the apache profile if necessary. Oh, and apache still runs in complain mode because my profiles are not complete yet (and because of bug 136651 which makes enforced access restrictions per vHost like a selfmade DOS). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |suse-beta@cboltz.de ------- Comment #9 from poeml@novell.com 2006-11-08 10:20 MST ------- Since I didn't receive your mail, could you please resend it? Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863
suse-beta@cboltz.de changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |ASSIGNED
Info Provider|suse-beta@cboltz.de |
------- Comment #10 from suse-beta@cboltz.de 2006-11-08 15:48 MST -------
Done.
Nov 8 23:47:24 server postfix/smtp[6412]: B245817790: to=
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #11 from poeml@novell.com 2006-11-08 16:05 MST ------- I have my doubts that this mail reaches me either. Can you send it to poeml@suse.de please? Thanks a million. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863
------- Comment #12 from suse-beta@cboltz.de 2006-11-09 10:08 MST -------
Done.
Nov 9 18:07:09 server postfix/smtp[26264]: 131F317790: to=
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #13 from poeml@novell.com 2006-11-10 16:21 MST ------- I looked at the config, and it looks sane. I read the bug again, and thought about it. Logically, the place where modules to be loaded are listed is /etc/apache2/sysconfig.d/loadmodule.conf, and the file is rewritten by rcapache2. Thus, writing the file could just be in progress, when another instance of rcapache2 kicks off an httpd process which reads the file. At least theoretically this could happen, even though I would expect the chance to be quite small. (Notwithstanding the fact, that I am not aware that logrotate runs jobs in background.) So I tried a little test. First, I started 10 rcapache2 in parallel, but it didn't reproduce the bug. In the second attempt I went higher, and with 100 jobs in parallel it was reproduced right away: -(~)-------------------------------------------------------------------------------------(root@batavia510)- `--# tail -f /var/log/apache2/error_log & [1] 31585 -(~)-------------------------------------------------------------------------------------(root@batavia510)- `--# for i in `seq 1 100`; do echo rcapache2 reload | at now + 1 min; done warning: commands will be executed using /bin/sh job 13 at 2006-11-11 00:07 warning: commands will be executed using /bin/sh job 14 at 2006-11-11 00:07 warning: commands will be executed using /bin/sh [many more of those...] warning: commands will be executed using /bin/sh job 111 at 2006-11-11 00:07 warning: commands will be executed using /bin/sh job 112 at 2006-11-11 00:07 -(~)-------------------------------------------------------------------------------------(root@batavia510)- `--# date Sat Nov 11 00:06:58 CET 2006 -(~)-------------------------------------------------------------------------------------(root@batavia510)- `--# [Sat Nov 11 00:07:01 2006] [notice] SIGUSR1 received. Doing graceful restart Syntax error on line 10 of /etc/apache2/mod_status.conf: Invalid command 'Order', perhaps misspelled or defined by a module not included in the server configuration -(~)-------------------------------------------------------------------------------------(root@batavia510)- `--# rcapache2 status Checking for httpd2: dead [2] 3840 exit 1 rcapache2 status -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #14 from poeml@novell.com 2006-11-10 16:40 MST ------- We'll have to investigate whether logrotate is working as designed, or whether it should serialize the script calls, and think about some way to synchronize rcapache2 runs. I don't like the multiple reloads anyway, that logrotate does... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #15 from suse-beta@cboltz.de 2006-11-10 17:09 MST ------- Wow. Indeed, this reproduces the problem. Thanks for finding the cause of this problem! I have added a workaround to /etc/init.d/apache2 and now can no longer reproduce the bug: --- apache2_orig 2006-11-11 00:38:25.000000000 +0100 +++ apache2 2006-11-11 00:52:37.000000000 +0100 @@ -69,6 +69,8 @@ } action="$1" +# workaround for #214863: don't regenerate loadmodule.conf on reload +test "$action" = "reload" && get_module_list_done=true case "$action" in stop|try-restart|*status*|probe) ;; Rewriting loadmodule.conf will need a "restart" now, but that's ways better than a crashing apache ;-) (In reply to comment #14)
We'll have to [...] think about some way to synchronize rcapache2 runs.
A suggestion how this could be done (untested, partly pseudocode, quoting missing etc. - but you should get the point) current code is create_loadmodule_conf > /etc/apache2/sysconfig.d/loadmodule.conf better code would be tmpfile=`mktemp apache.XXXXXX` create_loadmodule_conf > $tmpfile mv $tmpfile /etc/apache2/sysconfig.d/loadmodule.conf This way apache would not see a "half" loadmodule.conf. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #16 from suse-beta@cboltz.de 2006-11-12 07:54 MST ------- Another thing: loadmodule.conf should only be updated if /etc/sysconfig/apache2 is newer. test /etc/sysconfig/apache2 -nt /etc/apache2/sysconfig.d/loadmodule.conf && create_loadmodule_conf -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 ------- Comment #17 from poeml@novell.com 2006-12-20 09:06 MST ------- I like your suggestion from comment #14 (the atomic replace). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|poeml@novell.com |bnc-team-apache@forge.provo.novell.com Status|ASSIGNED |NEW -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|missing <IfModule> in |loadmodule.conf needs to be replaced atomically |/etc/apache2/server- | |tuning.conf and a strange | |apache exit | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863 poeml@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bnc-team-apache@forge.provo.novell.com AssignedTo|bnc-team- |skh@novell.com |apache@forge.provo.novell.co| |m | Status|ASSIGNED |NEW ------- Comment #18 from poeml@novell.com 2007-03-14 04:27 MST ------- Sonja will have a look -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214863#c19
--- Comment #19 from Christian Boltz
https://bugzilla.novell.com/show_bug.cgi?id=214863#c20
Peter Poeml
https://bugzilla.novell.com/show_bug.cgi?id=214863#c21
--- Comment #21 from Dirk Mueller
https://bugzilla.novell.com/show_bug.cgi?id=214863#c22
--- Comment #22 from Christian Boltz
I've replaced it by "sharedscripts" to avoid the multiple reload ..
This is only a workaround, not a solution... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=214863#c23
Sonja Krause-Harder
https://bugzilla.novell.com/show_bug.cgi?id=214863#c24
Dirk Mueller
participants (1)
-
bugzilla_noreply＠novell.com