[opensuse] Mysterious apache server crash, systemd seems clueless, and mod_python may (or may not) be the root cause.
(Current Tumbleweed, apache2-2.4.25-3.1.x86_64 - but has been happening since original install in January) Every couple of days (I have not located that part of the pattern), my apache server dies. The only pattern I see is that is it always around midnight when it happens. ● apache2.service - The Apache Webserver Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled) Active: failed (Result: core-dump) since Wed 2017-03-15 00:00:03 CET; 7h ago Process: 5490 ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop (code=exited, status=0/SUCCESS) Process: 5477 ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful (code=exited, status=0/SUCCESS) Process: 598 ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start (code=dumped, signal=ABRT) Main PID: 598 (code=dumped, signal=ABRT) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" CPU: 2min 58.455s Mar 14 12:59:40 sto-opq-src systemd[1]: Starting The Apache Webserver... Mar 14 12:59:40 sto-opq-src systemd[1]: Started The Apache Webserver. Mar 15 00:00:02 sto-opq-src systemd[1]: Reloading The Apache Webserver. Mar 15 00:00:02 sto-opq-src systemd[1]: Reloaded The Apache Webserver. Mar 15 00:00:03 sto-opq-src systemd[1]: apache2.service: Main process exited, code=dumped, status=6/ABRT Mar 15 00:00:03 sto-opq-src start_apache2[5490]: httpd (pid 598?) not running Mar 15 00:00:03 sto-opq-src systemd[1]: apache2.service: Unit entered failed state. Mar 15 00:00:03 sto-opq-src systemd[1]: apache2.service: Failed with result 'core-dump'. So, it looks like the server is restarting, and then 3 minutes later dumping core. I do not know why it is restarting. And having restarted, why it would die. It has never died when restarted via systemctrl (or rcapache2). Given the repeatable time, I suspected that a con job might be the cause. But /var/spool/cron has nothing at all. So perhaps systemctrl is running something? journalctl for about that time shows: Jan 14 00:00:02 sto-opq-src systemd[1]: Starting Rotate log files... Jan 14 00:00:02 sto-opq-src systemd[1]: Reloading The Apache Webserver. Jan 14 00:00:03 sto-opq-src systemd[1]: Reloaded The Apache Webserver. Jan 14 00:00:03 sto-opq-src systemd[1]: apache2.service: Main process exited, code=dumped, status=6/ABRT Jan 14 00:00:03 sto-opq-src start_apache2[5120]: httpd (pid 1730?) not running Jan 14 00:00:03 sto-opq-src systemd[1]: apache2.service: Unit entered failed state. Jan 14 00:00:03 sto-opq-src systemd[1]: apache2.service: Failed with result 'core-dump'. Jan 14 00:00:06 sto-opq-src systemd-coredump[5118]: Process 1730 (httpd-prefork) of user 0 dumped core. So, it looks like log rotation is happening when this happens. Could it be that some apache log is getting messed with so apache dies? I have not done anything with logs, so they are set however the system and various packages have set them. If I look at apache2's error_log, I see: [Wed Mar 15 00:00:03.011278 2017] [:notice] [pid 598] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads. [Wed Mar 15 00:00:03.011318 2017] [:notice] [pid 598] mod_python: using mutex_directory /tmp Fatal Python error: PyEval_SaveThread: NULL tstate [Wed Mar 15 00:00:03.011415 2017] [core:notice] [pid 598] AH00060: seg fault or similar nasty error detected in the parent process I'm not certain that mod_python is the cause. The apache2 server provides access to a Trac site (we love Trac: https://trac.edgewall.org/) . So we are big users of mod_python. I'm not sure how to proceed. Obviously the apache2 problem should be solved. Nonetheless, I would think systemd would ensure that the server is running. I do not see that this is happening. If it were restarted again I would see the restart in the log. I do not see this. 1. How to get systemd to restart apache2 again so it is runnung? 2. Anyone else using mod_python in apache2? -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 15.03.2017 08:20, Roger Oberholtzer wrote:
(Current Tumbleweed, apache2-2.4.25-3.1.x86_64 - but has been happening since original install in January)
Every couple of days (I have not located that part of the pattern), my apache server dies. The only pattern I see is that is it always around midnight when it happens.
...
I'm not certain that mod_python is the cause. The apache2 server provides access to a Trac site (we love Trac: https://trac.edgewall.org/) . So we are big users of mod_python.
I'm not sure how to proceed. Obviously the apache2 problem should be solved. Nonetheless, I would think systemd would ensure that the server is running. I do not see that this is happening. If it were restarted again I would see the restart in the log. I do not see this.
1. How to get systemd to restart apache2 again so it is runnung?
2. Anyone else using mod_python in apache2?
I have observed similar behavior (apache service dying, package version 2.4.23-6.1) after upgrading my home server from 13.2 to 42.2. The apache instance is running owncloud and a local OBS 2.7 server. AS i don't really have the time to dig into this, I have adapted my apache2 unit file to automatically restart apache in such cases, see below (the changes are below the comment lines. Unit file at: /usr/lib/systemd/system/apache2.service -----8<---------------------------------------------- [Unit] Description=The Apache Webserver After=network.target nss-lookup.target time-sync.target remote-fs.target Before=getty@tty1.service plymouth-quit.service xdm.service PartOf=apache2.target [Service] Type=notify PrivateTmp=true ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop KillMode=mixed TasksMax=infinity NotifyAccess=all # 2017-03-13 AM: added restart tag to work around the occasional crashes Restart=on-failure # 2017-03-13 AM: ... and wait for 30 seconds before performing the restart, just in case RestartSec=30 [Install] WantedBy=multi-user.target Alias=httpd.service apache.service -----8<---------------------------------------------- -- Cahn's Axiom: When all else fails, read the instructions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Mar 15, 2017 at 10:23 AM, Andreas Mahel
Roger Oberholtzer wrote:
On Wed, Mar 15, 2017 at 10:23 AM, Andreas Mahel
wrote: I added:
Restart=on-failure RestartSec=30
and ran:
systemctl daemon-reload systemctl restart apache2
I guess /usr/lib/systemd/system/apache2.service will be replaced if apache2 is updated?
You could have added a drop-in config file instead. /etc/systemd/system/apache2.service.d/dropin.conf btw, I googled "Fatal Python error: PyEval_SaveThread: NULL tstate" this morning and saw a couple of hits - it might be worth looking up. -- Per Jessen, Zürich (10.4°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 15.03.2017 11:46, Per Jessen wrote:
Roger Oberholtzer wrote:
I guess /usr/lib/systemd/system/apache2.service will be replaced if apache2 is updated?
You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
Cool, thanks for the pointer (once you know the correct naming, googling it gets a lot easier). -- Cahn's Axiom: When all else fails, read the instructions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andreas Mahel wrote:
On 15.03.2017 11:46, Per Jessen wrote:
Roger Oberholtzer wrote:
I guess /usr/lib/systemd/system/apache2.service will be replaced if apache2 is updated?
You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
Cool, thanks for the pointer (once you know the correct naming, googling it gets a lot easier).
Afterwards, "systemctl cat <service>" is very useful to see if your drop-in has been picked up. -- Per Jessen, Zürich (13.7°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Mar 15, 2017 at 2:13 PM, Per Jessen
You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
There is a link to the file I edited: /etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service So, would I remove that link and make the /etc/systemd/system/apache2.service.d/dropin.conf? Having both seems confusing. And will the link I remove be replaced when the package is updated? I know. Read the systemd manual... -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Roger Oberholtzer
On Wed, Mar 15, 2017 at 2:13 PM, Per Jessen
wrote: You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
There is a link to the file I edited:
/etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
So, would I remove that link and make the /etc/systemd/system/apache2.service.d/dropin.conf?
Having both seems confusing. And will the link I remove be replaced when the package is updated?
I know. Read the systemd manual...
no, just read the email: "You could have added a drop-in config file instead." restore the edited service file (un-edit), and add the "dropin.conf" file -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 Photos: http://wahoo.no-ip.org/piwigo @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Mar 15, 2017 at 4:20 PM, Patrick Shanahan
* Roger Oberholtzer
[03-15-17 10:57]: On Wed, Mar 15, 2017 at 2:13 PM, Per Jessen
wrote: You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
There is a link to the file I edited:
/etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
So, would I remove that link and make the /etc/systemd/system/apache2.service.d/dropin.conf?
Having both seems confusing. And will the link I remove be replaced when the package is updated?
I know. Read the systemd manual...
no, just read the email: "You could have added a drop-in config file instead."
Hmm. There currently exists a file with one name in the /etc/systemd/system directory. The suggestion was to add a directory containing a file with a different name. Perhaps the name in the directory just needs to end in .conf. But that still says nothing about the existing file. Remove it? Most likely. But as it was added by the apache2 package, will it get added again when the package is updated? After all, the package added the link that I will be removing. Or, if there is a apache2.service..d directory, is any file then ignored and only the contents of the apache2.service..d directory used? -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Roger Oberholtzer wrote:
Hmm. There currently exists a file with one name in the /etc/systemd/system directory. The suggestion was to add a directory containing a file with a different name. Perhaps the name in the directory just needs to end in .conf.
I think that is correct, yes.
But that still says nothing about the existing file. Remove it?
Nono, certainly not. The drop-in adds or replaces configs in the existing file.
Or, if there is a apache2.service..d directory, is any file then ignored and only the contents of the apache2.service..d directory used?
No, the regular apache service file is used, then amended by whatever you put in the drop in. -- Per Jessen, Zürich (16.0°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Mar 15, 2017 at 4:48 PM, Per Jessen
No, the regular apache service file is used, then amended by whatever you put in the drop in.
After adding this file, systemctl add the following: Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/apache2.service.d └─restart.conf So, seems like it is using the additional information. Next is to see that the server is always running. And to see if mod_python is the cause of the crash. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-03-15 16:48, Per Jessen wrote:
Roger Oberholtzer wrote:
Hmm. There currently exists a file with one name in the /etc/systemd/system directory. The suggestion was to add a directory containing a file with a different name. Perhaps the name in the directory just needs to end in .conf.
I think that is correct, yes.
But that still says nothing about the existing file. Remove it?
Nono, certainly not. The drop-in adds or replaces configs in the existing file.
He has: /etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service I think the one on /etc should be removed. If you edit the file in etc it makes sense to have it, but if both are the same, or a link, you should only have the one in /usr, which should be the original one, not edited. Maybe he edited the one in /usr, and the system retaliated by placing a link in /etc. Thus when there is an update, the one in /usr would be replaced, and perhaps the edited version would remain, the link broken, dunno. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))
Carlos E. R. wrote:
On 2017-03-15 16:48, Per Jessen wrote:
Roger Oberholtzer wrote:
Hmm. There currently exists a file with one name in the /etc/systemd/system directory. The suggestion was to add a directory containing a file with a different name. Perhaps the name in the directory just needs to end in .conf.
I think that is correct, yes.
But that still says nothing about the existing file. Remove it?
Nono, certainly not. The drop-in adds or replaces configs in the existing file.
He has:
/etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
I think the one on /etc should be removed. If you edit the file in etc it makes sense to have it, but if both are the same, or a link, you should only have the one in /usr, which should be the original one, not edited.
I think /etc/systemd/system/apache.service is just for convenience, to be able to use 'apache' instead of 'apache2'. You shouldn't edit any of them, link or otherwise, that's the point. -- Per Jessen, Zürich (15.0°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-03-16 14:33, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-03-15 16:48, Per Jessen wrote:
Roger Oberholtzer wrote:
Hmm. There currently exists a file with one name in the /etc/systemd/system directory. The suggestion was to add a directory containing a file with a different name. Perhaps the name in the directory just needs to end in .conf.
I think that is correct, yes.
But that still says nothing about the existing file. Remove it?
Nono, certainly not. The drop-in adds or replaces configs in the existing file.
He has:
/etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
I think the one on /etc should be removed. If you edit the file in etc it makes sense to have it, but if both are the same, or a link, you should only have the one in /usr, which should be the original one, not edited.
I think /etc/systemd/system/apache.service is just for convenience, to be able to use 'apache' instead of 'apache2'. You shouldn't edit any of them, link or otherwise, that's the point.
Ah, I didn't notice the absence of the "2". But there is an error here: files in /etc/systemd/system/ are intended to be edited. If this is an exception, the symlink should be /usr/lib/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service Or be removed completely. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))
Carlos E. R. wrote:
On 2017-03-16 14:33, Per Jessen wrote:
I think /etc/systemd/system/apache.service is just for convenience, to be able to use 'apache' instead of 'apache2'. You shouldn't edit any of them, link or otherwise, that's the point.
Ah, I didn't notice the absence of the "2".
But there is an error here: files in /etc/systemd/system/ are intended to be edited. If this is an exception, the symlink should be /usr/lib/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
Maybe - there are a few of them in /etc/systemd/system : apache.service httpd.service network.service syslog.service. -- Per Jessen, Zürich (15.7°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-03-16 15:03, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-03-16 14:33, Per Jessen wrote:
I think /etc/systemd/system/apache.service is just for convenience, to be able to use 'apache' instead of 'apache2'. You shouldn't edit any of them, link or otherwise, that's the point.
Ah, I didn't notice the absence of the "2".
But there is an error here: files in /etc/systemd/system/ are intended to be edited. If this is an exception, the symlink should be /usr/lib/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
Maybe - there are a few of them in /etc/systemd/system :
apache.service httpd.service network.service syslog.service.
Alias. Yes. Added by the distribution, not on upstream, I guess. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))
Roger Oberholtzer wrote:
On Wed, Mar 15, 2017 at 2:13 PM, Per Jessen
wrote: You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
There is a link to the file I edited:
/etc/systemd/system/apache.service -> /usr/lib/systemd/system/apache2.service
So, would I remove that link and make the /etc/systemd/system/apache2.service.d/dropin.conf?
No, just restore /usr/lib/systemd/system/apache2.service to whatever it was, and put your changes in the drop-in.
Having both seems confusing. And will the link I remove be replaced when the package is updated?
The drop-in will not be overwritten, that's the main thing.
I know. Read the systemd manual...
Nah, that's mailing lists are here for. :-) -- Per Jessen, Zürich (16.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-03-15 14:13, Per Jessen wrote:
Andreas Mahel wrote:
On 15.03.2017 11:46, Per Jessen wrote:
Roger Oberholtzer wrote:
I guess /usr/lib/systemd/system/apache2.service will be replaced if apache2 is updated?
You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
Cool, thanks for the pointer (once you know the correct naming, googling it gets a lot easier).
Afterwards, "systemctl cat <service>" is very useful to see if your drop-in has been picked up.
For my education, what would have that conf file? Just: Restart=on-failure RestartSec=30 and that's all? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))
On Thu, Mar 16, 2017 at 12:11 PM, Carlos E. R.
For my education, what would have that conf file? Just:
Restart=on-failure RestartSec=30
and that's all?
Yes. So I named it restart.conf, since that is what it should do. The original apache2.services file is unmodified. I think the main thing to keep in mind is that the directory should be named as the service is. I do not think it can be named as an alias. I have not tried that. If it could be named for the alias, then I guess you could easily make variations on the original service with the differences in a per-alias subdirectory. Then start and stop the service with the appropriate alias to get the exact settings you want. I need a clone (of me...). Too little time to try everything! -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2017-03-15 14:13, Per Jessen wrote:
Andreas Mahel wrote:
On 15.03.2017 11:46, Per Jessen wrote:
Roger Oberholtzer wrote:
I guess /usr/lib/systemd/system/apache2.service will be replaced if apache2 is updated?
You could have added a drop-in config file instead.
/etc/systemd/system/apache2.service.d/dropin.conf
Cool, thanks for the pointer (once you know the correct naming, googling it gets a lot easier).
Afterwards, "systemctl cat <service>" is very useful to see if your drop-in has been picked up.
For my education, what would have that conf file? Just:
Restart=on-failure RestartSec=30
and that's all?
Yep, that's all. Only the extra config directives you want. -- Per Jessen, Zürich (12.4°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 16.03.2017 12:11, Carlos E. R. wrote:
For my education, what would have that conf file? Just:
Restart=on-failure RestartSec=30
and that's all?
You would need the section as well: [Service] Restart=on-failure RestartSec=30 -- Cahn's Axiom: When all else fails, read the instructions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Mar 16, 2017 at 1:01 PM, Andreas Mahel
You would need the section as well:
[Service] Restart=on-failure RestartSec=30
I did have that in the file. I was perhaps too quick to answer the question. Sorry. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Andreas Mahel
-
Carlos E. R.
-
Patrick Shanahan
-
Per Jessen
-
Roger Oberholtzer